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Abstract 

The  purpose  of  this  research  is  to  characterize  the  number  and  type  of  problems  satellite 
operators  encounter  during  the  course  of  routine  daily  activities.  In  the  context  of  this  paper, 
a  discrepancy  is  defined  as  the  perception  by  an  operator  that  some  portion  of  the  space 
system  has  failed  to  operate  as  designed.  Discrepancies  can  therefore  include  ground  system 
malfunctions,  procedural  errors,  and  even  misdiagnosis  that  a  problem  exists,  in  addition  to 
actual  spacecraft  anomalies.  The  study  is  designed  to  test  the  following  hypotheses  using  a 
verifiable,  quantitative  analysis:  1)  the  majority  of  problems  encountered  by  an  operator  do 
not  involve  the  spacecraft  at  all,  but  are  attributed  to  other  elements  of  the  space  system;  and 
2)  correlations  exist  between  aspects  of  a  space  system  design  and  the  nature  of  problems 
experienced  by  the  operations  staff  over  the  long  term. 

To  perform  an  applied,  rather  than  theoretical,  analysis,  several  representative  aerospace 
organizations  provided  historical  discrepancy  reports  for  satellites  currently  being  flown. 
Government  civilian  and  military  organizations  participated  in  the  study,  which  encompassed 
spacecraft  from  several  mission  areas,  including  remote  sensing,  research  &  development, 
communications,  and  observatories.  Raw  discrepancy  data  was  collected  from  each 
organization  in  the  form  of  electronic  databases  or  hand-written  logs.  Data  conversion 
consisted  of  selecting  similar  fields  of  interest  from  each  database;  removing  references  to 
the  program,  organization,  or  spacecraft;  replacing  program-specific  terminology  with 
standardized  nomenclature;  and  merging  the  individual  databases  into  one  data  set. 

Statistical  analysis  of  the  entire  data  set  was  performed  to  test  the  two  hypotheses  above. 

The  results  of  the  analysis  found  that  13%  of  all  discrepancies  reported  are  associated  with 
subsystems  on  board  the  spacecraft,  while  the  remaining  87%  are  attributed  to  the  operations 
infrastructure,  including  ground  systems,  communications  relays,  facilities,  and  associated 
equipment.  Correlations  were  found  between  certain  design  characteristics  and  the  types  of 
discrepancies  reported  for  a  given  mission.  In  addition,  software  problems  were  the  root 
cause  in  an  average  48%  of  the  discrepancies  reported  for  each  mission. 
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The  purpose  of  this  research  is  to  characterize  the  number  and  type  of  problems  satellite 
operators  encounter  during  the  course  of  routine  daily  activities.  In  the  context  of  this  paper, 
a  discrepancy  is  defined  as  the  perception  by  an  operator  that  some  portion  of  the  space 
system  has  failed  to  operate  as  designed.  Discrepancies  can  therefore  include  ground  system 
malfunctions,  procedural  errors,  and  even  misdiagnosis  that  a  problem  exists,  in  addition  to 
actual  spacecraft  anomalies.  The  study  is  designed  to  test  the  following  hypotheses  using  a 
verifiable,  quantitative  analysis:  1)  the  majority  of  problems  encountered  by  an  operator  do 
not  involve  the  spacecraft  at  all,  but  are  attributed  to  other  elements  of  the  space  system;  and 
2)  correlations  exist  between  aspects  of  a  space  system  design  and  the  nature  of  problems 
experienced  by  the  operations  staff  over  the  long  term. 

To  perform  an  applied,  rather  than  theoretical,  analysis,  several  representative  aerospace 
organizations  provided  historical  discrepancy  reports  for  satellites  currently  being  flown. 
Government  civilian  and  military  organizations  participated  in  the  study,  which  encompassed 
spacecraft  from  several  mission  areas,  including  remote  sensing,  research  &  development, 
communications,  and  observatories.  Raw  discrepancy  data  was  collected  from  each 
organization  in  the  form  of  electronic  databases  or  hand-written  logs.  Data  conversion 
consisted  of  selecting  similar  fields  of  interest  from  each  database;  removing  references  to 
the  program,  organization,  or  spacecraft;  replacing  program-specific  terminology  with 
standardized  nomenclature;  and  merging  the  individual  databases  into  one  data  set. 

Statistical  analysis  of  the  entire  data  set  was  performed  to  test  the  two  hypotheses  above. 

The  results  of  the  analysis  found  that  13%  of  all  discrepancies  reported  are  associated  with 
subsystems  on  board  the  spacecraft,  while  the  remaining  87%  are  attributed  to  the  operations 
infrastructure,  including  ground  systems,  communications  relays,  facilities,  and  associated 
equipment.  Correlations  were  found  between  certain  design  characteristics  and  the  types  of 
discrepancies  reported  for  a  given  mission.  In  addition,  software  problems  were  the  root 
cause  in  an  average  48%  of  the  discrepancies  reported  for  each  mission. 
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Abstract 

The  purpose  of  this  research  is  to  characterize  the  number  and  type  of  problems  satellite 
operators  encounter  during  the  course  of  routine  daily  activities.  In  the  context  of  this  paper, 
a  discrepancy  is  defined  as  the  perception  by  an  operator  that  some  portion  of  the  space 
system  has  failed  to  operate  as  designed.  Discrepancies  can  therefore  include  ground  system 
malfunctions,  procedural  errors,  and  even  misdiagnosis  that  a  problem  exists,  in  addition  to 
actual  spacecraft  anomalies.  The  study  is  designed  to  test  the  following  hypotheses  using  a 
verifiable,  quantitative  analysis:  1)  the  majority  of  problems  encountered  by  an  operator  do 
not  involve  the  spacecraft  at  all,  but  are  attributed  to  other  elements  of  the  space  system;  and 
2)  correlations  exist  between  aspects  of  a  space  system  design  and  the  nature  of  problems 
experienced  by  the  operations  staff  over  the  long  term. 

To  perform  an  applied,  rather  than  theoretical,  analysis,  several  representative  aerospace 
organizations  provided  historical  discrepancy  reports  for  satellites  currently  being  flown. 
Government  civilian  and  military  organizations  participated  in  the  study,  which  encompassed 
spacecraft  from  several  mission  areas,  including  remote  sensing,  research  &  development, 
communications,  and  observatories.  Raw  discrepancy  data  was  collected  from  each 
organization  in  the  form  of  electronic  databases  or  hand-written  logs.  Data  conversion 
consisted  of  selecting  similar  fields  of  interest  from  each  database;  removing  references  to 
the  program,  organization,  or  spacecraft;  replacing  program-specific  terminology  with 
standardized  nomenclature;  and  merging  the  individual  databases  into  one  data  set. 

Statistical  analysis  of  the  entire  data  set  was  performed  to  test  the  two  hypotheses  above. 

The  results  of  the  analysis  found  that  13%  of  all  discrepancies  reported  are  associated  with 
subsystems  on  board  the  spacecraft,  while  the  remaining  87%  are  attributed  to  the  operations 
infrastructure,  including  ground  systems,  communications  relays,  facilities,  and  associated 
equipment.  Correlations  were  found  between  certain  design  characteristics  and  the  types  of 
discrepancies  reported  for  a  given  mission.  In  addition,  software  problems  were  the  root 
cause  in  an  average  48%  of  the  discrepancies  reported  for  each  mission. 
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Executive  Summary 


Introduction 

The  purpose  of  this  research  is  to  characterize  the  number  and  type  of  problems  satellite 
operators  encounter  during  the  course  of  routine  daily  activities.  Problems  -  regardless  of 
cause,  associated  subsystem,  and  effect  -  are  a  source  of  uncertainty  and  cost  in  satellite 
operations.  Spacecraft  designers,  integrators,  and  operators  take  extraordinary  precautions  to 
avoid  defects  once  a  spacecraft  is  on-orbit.  Yet  despite  all  the  care  and  attention  devoted  to 
this  cause,  problems  of  all  types  still  occur. 

Unfortunately,  there  is  a  notable  lack  of  published  data  concerning  the  day-to-day  issues 
affecting  satellite  operations.  What  types  of  problems  will  a  satellite  operator  encounter 
during  a  typical  week,  month,  or  year  of  work?  How  often  will  these  problems  occur? 
Although  experienced  members  of  an  operations  staff  can  usually  provide  estimates  based  on 
anecdotal  evidence,  there  exists  no  precedent  for  making  this  information  available  to  the 
aerospace  community  at  large. 

Several  studies  have  documented  the  failure  history  of  spacecraft  components,  but  these 
studies  do  not  capture  the  full  scope  of  problems  that  an  operator  might  face.  On  a  routine 
daily  basis,  satellite  operators  may  encounter  problems  completely  unrelated  to  the  spacecraft 
and  unaffected  by  its  performance.  Lifecycle  operations  play  an  important  role  in  a  typical 
space  system  enterprise,  and  one  that  is  sometimes  neglected  or  ignored.  For  satellites  with  a 
long  design  life  or  a  particularly  complex  ops  concept,  the  operations  phase  may  easily 
surpass  the  design  and  manufacturing  stages  in  terms  of  time  and  cost.  Therefore,  all  the 
problems  encountered  during  the  operations  phase  -  including  those  not  related  to  the 
spacecraft  -  should  be  documented,  hence  the  motivation  for  this  study. 

The  study  is  designed  to  test  the  following  hypotheses  using  a  verifiable,  quantitative 
analysis: 
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•  First  hypothesis.  Most  problems  encountered  by  an  operator  do  not  involve  the 
spacecraft  at  all,  but  are  attributed  to  other  elements  of  the  space  system. 

•  Second  hypothesis.  Correlations  exist  between  aspects  of  a  space  system  design  and 
the  nature  of  problems  experienced  by  the  operations  staff  over  the  long  term. 

In  the  context  of  this  paper,  a  discrepancy  is  defined  as  the  perception  by  an  operator  that 
some  portion  of  the  space  system  has  failed  to  operate  as  designed.  Discrepancies  can 
therefore  include  ground  system  malfunctions,  procedural  errors,  and  even  misdiagnosis  that 
a  problem  exists,  in  addition  to  actual  spacecraft  anomalies. 

Methodology 

Several  government  civilian  and  military  satellite  programs  were  contacted  about 
contributing  discrepancy  data  for  this  study.  Although  the  names  and  formats  of  the  reports 
varied  greatly,  all  of  the  programs  maintained  some  form  of  documentation  for  the  problems 
they  encountered.  The  documentation  targeted  for  this  study  was  the  first-response  problem 
logging  or  reporting  performed  by  the  on-console  operators,  if  available.  When  provided  in 
the  form  of  hand -written  logs,  individual  entries  were  first  manually  entered  into  electronic 
format.  Hardcopy  forms  and  incident  reports  were  scanned  using  optical  character 
recognition  (OCR)  software  and  reviewed  for  accuracy.  Source  data  provided  in  the  form  of 
proprietary  or  mission-specific  electronic  databases  were  first  exported  to  tab-delimited  text 
files.  In  each  of  the  three  cases,  the  result  was  a  generic  text  file  containing  all  the  original 
source  data,  which  could  be  manipulated  by  a  variety  of  software  tools. 

All  of  the  source  data  sets  used  a  unique  format  in  terms  of  structure,  data  requested,  field 
names,  etc.  In  order  merge  the  data  into  one  uniform  database,  several  modifications  were 
made  to  each  set.  To  begin  with,  a  standard  data  format  was  defined  using  a  core  set  of 
information  common  to  a  large  majority  of  the  source  sets.  Unnecessary  or  extraneous  data 
fields  were  removed  from  the  source  data  sets,  and  the  pseudonym  for  the  contributing 
organization  was  added  to  every  entry  in  the  file. 
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The  next  data  preparation  step  was  accomplished  to  address  concerns  about  the  release  of 
proprietary  data.  References  to  spacecraft  names,  payload  instruments,  personnel  teams, 
unique  facilities,  or  specific  equipment  configurations  were  replaced  with  generic  descriptive 
terms.  At  this  point  in  the  process,  the  discrepancy  reports  were  normalized  on  the  basis  of 
each  spacecraft.  This  was  accomplished  by  dividing  the  number  of  discrepancies  associated 
with  each  subsystem  or  cause  for  a  given  spacecraft  by  the  sum  total  of  the  spacecraft’s 
discrepancies.  The  resulting  data  observations  for  the  spacecraft  were  a  set  of  percentages, 
one  for  each  subsystem  or  cause,  which  described  the  distribution  of  problems  among  the 
various  categories. 

Results 

Figure  1  shows  the  percentage  of  all  reported  discrepancies  that  belong  to  each  subsystem 
category. 


Figure  1  -  Percentage  of  All  Discrepancies  Reported  vs.  Associated  Subsystem 
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The  diagram  indicates  that  the  two  subsystems  most  frequently  associated  with  discrepancies 
were  ground  systems  and  communications  relays.  The  result  of  the  normalization  process  is 
a  set  of  percentages  representing  the  fraction  of  discrepancies  for  each  spacecraft  that  are 
associated  with  each  subsystem  category.  The  percentages  can  be  analyzed  to  determine 
whether  or  not  a  particular  subsystem  is  consistently  identified  as  a  problem  across  multiple 
spacecraft  programs.  Figure  2  gives  a  more  accurate  presentation  of  the  fact  that  ground 
systems,  in  particular,  and  to  some  extent  communications  relays  are  consistently  attributed 
to  discrepancies  across  several  different  spacecraft  programs. 


Figure  2  -  Box  Plot  of  Subsystem  Statistics 


Most  of  the  discrepancy  data  sets  contributed  for  this  study  contained  root  cause  information 
in  each  report.  Those  that  did  not  were  excluded  from  this  portion  of  the  analysis.  In  a 
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similar  fashion  as  the  subsystem  category,  the  first  step  is  to  report  the  number  of 
discrepancies  belonging  to  each  subsystem  category.  This  is  accomplished  with  the  category 
relative  frequency  diagram  shown  in  Figure  3.  The  diagram  indicates  that  software  problems 
are  the  most  frequent  cause  of  discrepancies,  occurring  in  61%  of  the  reported  cases.  The 
equipment,  unknown,  procedure,  and  no  anomaly  categories  were  comparable  at 
approximately  7%  each. 


Figure  3  -  Percentage  of  All  Discrepancies  Reported  vs.  Root  Cause 
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The  normalization  process  can  also  be  applied  to  the  root  cause  analysis  to  prevent  one  or 
two  spacecraft  from  skewing  the  results.  When  the  data  is  normalized  on  a  per-spacecraft 
basis,  the  resulting  statistical  parameters  for  each  root  cause  category  are  shown  graphically 
using  a  box  plot  in  Figure  4.  The  diagram  shows  that  software  is  the  only  root  cause 
category  consistently  reported  as  a  frequently  occurring  problem  among  all  of  the  spacecraft 
in  the  dataset. 
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Figure  4  -  Box  Plot  of  Root  Cause  Statistics 


Root  Cause 


Conclusions 

An  analysis  of  over  9,200  discrepancy  reports  from  1 1  on-orbit  spacecraft  supports  the  first 
hypothesis  -  87%  of  the  discrepancies  collected  were  attributed  to  some  component  of  the 
operational  infrastructure.  The  remaining  13%  involved  one  or  more  components  on  board 
the  spacecraft.  Software  was  the  most  frequently  reported  cause  of  discrepancies,  found  in 
61%  of  all  discrepancies  documented. 

The  discrepancy  reports  also  indicated  that  correlations  do  exist  between  certain  design 
elements  and  the  types  of  problems  experienced  during  operations.  The  following 
correlations  were  found  based  on  the  data  collected: 
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•  Ground  System  vs.  Mission  Type.  The  percentage  of  discrepancies  per  spacecraft 
associated  with  the  ground  system  tends  to  change  given  a  particular  mission  type  for 
the  spacecraft. 

•  Comm  Relay  vs.  Ops  Team.  The  percentage  of  discrepancies  per  spacecraft 
associated  with  the  communications  relay  tends  to  change  from  one  organization  to 
another. 

Thus,  the  data  collected  supports  the  second  hypothesis,  but  with  the  caveat  that  a  sufficiently 
large  and  diverse  sample  set  must  be  obtained  to  verify  the  results.  It  should  be  noted  that 
causality  cannot  be  determined  from  the  statistical  correlation  analysis,  but  must  be 
investigated  on  a  case-by-case  basis. 

The  results  of  this  study  can  be  extended  by  incorporating  discrepancy  data  from  additional 
spacecraft,  particularly  commercial  programs.  The  methodology  can  also  be  applied  on 
databases  for  individual  satellite  programs  to  gain  insight  into  the  nature  and  frequency  of 
problems  experienced  by  the  operations  staff.  Ultimately,  this  can  help  supervisors  identify 
strengths  and  areas  for  improvement  in  attempt  to  continuously  improve  the  service  provided 
to  the  user. 
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There  exists  a  great  deal  of  literature  on  the  theory  and  application  of  lean  principles.  The 
purpose  of  this  section  is  neither  to  expand  upon  existing  theory  nor  to  serve  as  an  exhaustive 
summary  of  available  literature.  It  is  intended  to  provide  only  a  very  brief  introduction  to 
lean  principles  so  the  reader  will  understand  the  context  in  which  this  research  was 
conducted.  The  book  Lean  Thinking,  by  James  Womack  and  Daniel  Jones,  contains  a  much 
more  thorough  treatment  of  this  subject  and  is  generally  considered  the  best  source  on  the 
background  and  application  of  lean  principles.  Thus,  this  chapter  simply  recounts  the  origins 
of  lean,  the  creation  of  the  Lean  Aerospace  Initiative  (LAI),  and  the  underlying  motivation 
for  operational  satellite  discrepancy  research. 

Origins  of  Lean 

The  origins  of  current  theory  on  lean  principles  can  be  traced  back  to  manufacturing 
techniques  implemented  at  Toyota  Motor  Company  following  World  War  II.  In  the  years 
immediately  following  the  war,  executives  at  Toyota  searched  for  ways  to  catch  up  to  the 
manufacturing  capabilities  of  their  American  counterparts.  The  task  and  responsibility  fell 
primarily  on  Toyota’s  chief  production  engineer,  Taiichi  Ohno.  After  several  fact-finding 
trips  to  American  auto  factories,  Ohno  eventually  realized  that  Japan  simply  did  not  have  the 
resources  or  the  market  base  to  sustain  the  tremendous  level  of  mass  production  taking  place 
in  the  United  States.  What  he  needed  to  find,  instead,  was  a  more  efficient  way  to  deliver  the 
exact  products  his  customers  wanted  as  quickly  and  as  cheaply  as  possible.1  The  various 
manufacturing  techniques  he  developed,  and  more  importantly,  the  process  by  which  he 
developed  them,  collectively  form  the  basis  of  the  management  theory  now  known  as  ‘lean.’ 

Womack  and  Jones  summarize  Ohno’s  approach  using  five  principles  of  lean: 


1  Ohno,  Taiichi.  Toyota  Production  System:  Beyond  Large-Scale  Production .  Cambridge,  MA:  Productivity 
Press,  1988.  pp.  1-3. 

2  Womack,  James  P.  and  Daniel  T.  Jones.  Lean  Thinking:  Banish  Waste  and  Create  Wealth  in  Your 
Corporation .  New  York,  NY:  Simon  &  Schuster,  1996.  pp.  15-98. 
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•  Value.  The  first  and  critical  step  is  defining  what  is  important,  and  therefore  has 
value,  from  the  perspective  of  the  end  customer.  It  may  be  a  product,  a  service,  or  a 
combination  of  the  two.  The  definition  of  value  should  include  not  only  what  the 
customer  wants,  but  also  how  it  should  be  provided,  when  it  should  be  provided,  and 
at  what  cost. 

•  The  Value  Stream.  Once  the  concept  of  value  is  defined,  every  activity  -  from  the 
collection  of  raw  materials  to  the  delivery,  use,  and  maintenance  of  the  final  product  - 
must  be  examined  and  categorized  as  value-added,  wasteful  but  necessary  given 
current  constraints,  or  wasteful  and  unnecessary.  Value-added  activities  directly 
contribute  to  the  form,  fit,  and  function  of  a  product  or  the  successful  performance  of 
a  service.  Wasteful  activities  generally  can  be  categorized  as  one  of  seven  types: 
over-production,  waiting,  transportation,  inventory,  processing,  motion,  and  defects. 
The  goal  is  to  eventually  completely  eliminate  waste  from  the  value  stream. 

•  Flow.  Once  the  value  stream  is  thoroughly  mapped  and  as  much  waste  as  possible 
eliminated,  the  remaining  value-added  activities  must  be  organized  to  allow 
continuous  flow.  Ohno’s  implementation  focused  on  single-piece  flow  through  the 
assembly  line,  as  opposed  to  the  batch-and-queue  method.  He  achieved  this  using 
production  cells  capable  of  accomplishing  their  tasks  according  to  a  predefined  takt 
time.  The  recognition  of  time  as  a  critical  commodity,  and  the  emphasis  on 
synchronizing  work  in  a  precise  fashion,  is  one  of  the  distinguishing  characteristics  of 
lean.  Ohno  also  relied  on  personnel  who  were  organized  along  product  lines  or 
processes,  rather  than  functions  or  departments. 

•  Pull.  Continuous  flow  makes  it  possible  for  a  customer  to  ‘pull’  products  or  services 
from  a  company,  rather  than  a  company  pushing  products  or  services  onto  the 
customer.  This  creates  a  ripple  effect  that  reaches  all  the  way  back  through  the  value 
stream  to  the  original  suppliers.  The  combination  of  continuous  single-piece  flow 
and  pull  is  sometimes  called  just-in-time  manufacturing,  which  requires  an  ability  to 
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rapidly  changeover  from  one  product  to  another  and  to  produce  parts  with  a  defect 
rate  near  zero. 

•  Perfection.  The  initial  implementation  of  the  first  four  principles  is  not  enough. 

They  must  be  accomplished  repeatedly  in  an  iterative  fashion,  always  looking  for 
ways  to  improve  the  product  or  service  and  eliminate  waste.  This  step  tends  to 
capitalize  on  the  knowledge  and  expertise  of  the  workers  actually  performing  the 
tasks  on  a  daily  basis. 

Even  more  important  than  the  techniques  Ohno  developed  is  the  process  he  used  to 
implement  and  perfect  them.  These  radical  manufacturing  improvements  did  not  occur 
overnight.  In  fact,  it  took  Toyota  almost  thirty  years  to  tweak  and  fine  tune  its  manufacturing 
methods,  but  the  result  was  an  assembly  line  that,  by  the  mid-1980s,  easily  led  in  the  world  in 
terms  of  efficiency,  flexibility,  and  quality.3  Success  was  achieved  by  taking  advantage  of 
workers’  knowledge  and  expertise  -  and  encouraging  them  to  suggest  and  test  improvements 
in  a  controlled,  measurable,  and  repeatable  fashion.4 

After  losing  tremendous  amounts  of  market  share  and  profit  to  the  Japanese  in  the  1980s, 
American  automotive  companies  forced  themselves  to  learn  the  principles  of  the  Toyota 
Production  System.  Ford  Motor  Company  purchased  a  24%  share  of  Mazda  and  set  about 
learning  as  much  as  possible  from  the  company’s  production  complex  in  Hiroshima.5  GM 
launched  a  joint  venture  with  Toyota,  called  NUMMI,  which  was  located  in  California.  In 
both  cases,  the  American  companies  began  to  catch  up  with  their  Japanese  counterparts.  By 
1989,  American-owned  auto  factories  in  North  America  were  approaching  the  level  of 
productivity  found  in  Japanese-owned  plants. 


3  Womack,  James  P.,  Daniel  T.  Jones,  and  Daniel  Roos.  The  Machine  That  Changed  the  World:  The  Story  of 
Lean  Production.  New  York,  NY:  Harper  Perennial,  1991.  pp.  75-103. 

4  Spear,  Steven  and  H.  Kent  Bowen.  “Decoding  the  DNA  of  the  Toyota  Production  System,”  Harvard  Business 
Review.  (Sept-Oct)  Case  #  99509.  Cambridge,  M A:  Harvard  Business  School  Publishing,  1999.  pp.  97-106. 

5  Womack,  James  P.,  Daniel  T.  Jones,  and  Daniel  Roos.  The  Machine  That  Changed  the  World:  The  Story  of 
Lean  Production.  New  York,  NY:  Harper  Perennial,  1991.  pp.  237-238. 
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Creation  of  the  Lean  Aerospace  Initiative 

At  about  the  same  time  American  automotive  companies  began  to  realize  the  benefits  of 
converting  to  lean  production,  key  players  in  the  Department  of  Defense  and  industry  began 
to  question  whether  or  not  it  would  be  possible  to  apply  those  same  lessons  to  the  aerospace 
sector.  The  Lean  Aircraft  Initiative  was  created  in  1993  as  a  research  partnership  between 
the  U.S.  Air  Force,  the  Massachusetts  Institute  of  Technology,  defense  aerospace  companies, 
and  labor  unions.  Its  original  charter  was  to  examine  how  the  lean  principles  could  be 
applied  to  aircraft  manufacturing,  but  with  the  addition  of  several  other  government  agencies 
and  companies  by  1998,  the  scope  of  the  Lean  Aircraft  Initiative  was  broadened  to  include 
spacecraft  and  its  name  changed  to  the  Lean  Aerospace  Initiative  (LAI). 

The  Lean  Enterprise  Model  (LEM)  is  one  of  the  first  and  most  significant  products  developed 
by  LAI.  Introduced  in  June  1996,  the  LEM  is  a  systematic  framework  for  organizing  and 
disseminating  MIT  research  and  external  data  source  results.6  It  contains  an  enterprise-level 
summary  of  lean  principles  and  overarching  practices,  as  well  as  underlying  metrics  that  are 
useful  for  quantifying  and  measuring  various  aspects  of  lean  performance.  The  LEM  helps 
members  of  the  aerospace  community  find  research  related  to  particular  problem  areas,  and 
can  be  used  as  a  reference  tool  for  a  lean  self-assessment. 

This  research  is  sponsored  by  the  Lean  Aerospace  Initiative  and  applies  to  the  following 
areas  of  the  LEM:7 

•  Identify  and  Optimize  Enterprise  Flow.  In  order  to  optimize  the  flow  of  services 
provided  by  the  satellite  operations  community,  any  deviations  from  normal  operating 
conditions  must  be  identified.  Discrepancies,  which  represent  such  deviations,  must 
therefore  be  characterized. 

•  Ensure  Process  Capability  and  Maturation.  Once  the  deviations  from  normal 
operating  conditions  have  been  identified,  steps  must  be  taken  to  eliminate  them 


6  Lean  Aerospace  Initiative.  “Welcome  to  the  Lean  Enterprise  Model.”  Online  reference, 
http://lean.mit.edu/newleni2/.  March  15,  2001. 

7  Lean  Aerospace  Initiative,  “The  Lean  Enterprise  Model.”  Summary  Chart  with  Enabling  Practices. 
Cambridge,  MA:  Massachusetts  Institute  of  Technology.  July  1998. 
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wherever  possible.  Existing  satellite  operations  can  be  improved  by  taking  action  to 
correct  the  most  frequently  occurring  discrepancies. 

•  Continuously  Focus  on  the  Customer.  To  prevent  the  same  types  of  problems  from 
occurring  over  and  over  again,  future  satellite  programs  can  take  advantage  of  these 
lessons  to  incorporate  corrective  measures  directly  into  the  design. 


Motivation  for  Research 

Early  in  the  1990s,  lean  theory  expanded  from  its  origins  as  a  production  management 
method  to  include  the  concept  of  the  lean  enterprise.  Companies  in  the  process  of 
implementing  lean  principles  realized  that  in  order  to  reap  the  full  rewards  of  lean,  every 
aspect  of  their  business  -  not  just  the  factory  floor  -  had  to  be  improved.  It  became 
necessary  to  apply  lean  principles  to  business  areas  as  diverse  as  purchasing,  accounting, 
sales,  shipping,  logistics,  and  support.  This  extended  all  the  way  from  the  roots  of  the 
supplier  network  through  the  long-term  use  and  maintenance  of  the  product  by  the  customer. 

The  aerospace  sector  is  no  exception  to  this  rule.  For  a  typical  space  system,  the  entire 
enterprise  includes  an  acquisition  or  procurement  branch,  the  integrating  organization  and  its 
subcontractors  and  suppliers,  the  launch  system,  the  operating  organization  and  its 
infrastructure,  and  the  various  users  of  the  system  or  data.  Lifecycle  operations  play  an 
important  role  in  the  enterprise,  and  one  that  is  sometimes  neglected  or  ignored.  For 
satellites  with  a  long  design  life  or  a  particularly  complex  ops  concept,  the  operations  phase 
may  easily  surpass  the  design  and  manufacturing  stages  in  terms  of  time  and  cost.  Therefore, 
any  thorough  application  of  lean  principles  to  a  space  system  must  include  the  operations 
phase,  hence  the  motivation  for  this  study. 

The  purpose  of  this  research  is  to  characterize  the  number  and  type  of  problems  satellite 
operators  encounter  during  the  course  of  routine  daily  activities.  Chapter  2  provides 
background  on  the  need  for  discrepancy  research  within  the  aerospace  community.  Chapter 
3  highlights  the  hypotheses  tested  in  this  thesis  and  provides  an  overview  of  the  research 
structure.  Chapter  4  details  the  characterization  of  discrepancies,  while  Chapter  5  outlines 
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the  statistical  methods  used  to  analyze  them.  The  results  of  the  analyses  are  presented  in 
Chapter  6,  with  final  discussion  and  conclusions  in  Chapter  7. 

LAI  researcher  Annalisa  Weigel  laid  the  foundation  for  this  paper  with  her  study  of  satellite 
manufacturing  discrepancies  identified  during  the  system  integration  and  test  phase  of 
factory  operations.8  The  methods  and  analyses  described  in  Chapter  5  are  kept  consistent 
wherever  possible,  to  allow  easy  and  accurate  comparison  of  results. 


8  Weigel,  Annalisa.  “Spacecraft  System-Level  Integration  and  Test  Discrepancies:  Characterizing  Distributions 
and  Costs.”  Thesis.  Cambridge,  MA:  Massachusetts  Institute  of  Technology,  2000. 
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Chapter  2  -  Background  on  Discrepancies 

Problems  -  regardless  of  cause,  associated  subsystem,  and  effect  -  are  a  tremendous  source 
of  uncertainty  and  cost  in  satellite  operations.  Spacecraft  designers  add  redundant 
subsystems  to  minimize  the  impact  of  problems  that  will  inevitably  occur  over  the  lifetime  of 
the  system.  Operations  concepts  are  designed  to  prevent  the  possibility  of  obvious  single 
point  failures.  Operators  take  methodically  planned  precautionary  measures  to  avoid  even 
the  slightest  potential  for  problems.  Every  action  is  taken  with  the  knowledge  that  one 
seemingly  trivial  mistake  can  lead  to  the  catastrophic  failure  of  a  multi-million  dollar 
spacecraft.  Yet  despite  all  the  care  and  attention  devoted  to  space  systems,  problems  of  all 
types  still  occur. 

Unfortunately,  there  is  a  notable  lack  of  published  data  concerning  the  day-to-day  issues 
affecting  satellite  operations.  What  types  of  problems  will  a  satellite  operator  encounter 
during  a  typical  week,  month,  or  year  of  work?  How  often  will  these  problems  occur? 
Although  experienced  members  of  an  operations  staff  can  usually  provide  estimates  based  on 
anecdotal  evidence,  there  exists  no  precedent  for  making  this  information  available  to  the 
aerospace  community  at  large.  This  chapter  outlines  several  reasons  for  the  lack  of  detailed 
discrepancy  data,  examines  the  corresponding  impacts  of  the  lack  of  data,  and  summarizes 
the  potential  applications  of  discrepancy  data  once  it  is  made  available. 

Reasons  for  the  Lack  of  Discrepancy  Data 

There  are  several  reasons  why  satellite  operations  organizations  do  not  publish  discrepancy 
data.  Although  the  specific  reasons  vary  from  one  organization  to  another,  and  from  one 
individual  to  another,  they  are  generally  comprised  of  some  combination  of  the  reasons 
outlined  below: 

•  Confidentiality.  Many  companies  fear  that  releasing  discrepancy  data  reveals 

proprietary  information  about  the  way  they  conduct  operations  and  the  tools  they  use 
to  do  the  job.  Some  feel  that  revealing  this  information  would  even  cause  a  loss  of 
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competitive  advantage.  This  is  in  contrast  to  the  practice  of  releasing  status  and 
anomaly  information  about  the  spacecraft  itself,  which  users  have  a  legitimate  right  to 
know.  Military  organizations  have  a  similar  issue  in  that  most  discrepancy  data  is 
automatically  classified  on  the  grounds  it  reveals  too  much  information  about  the 
state  of  military  readiness,  or  more  importantly,  lack  thereof.  The  same  is  true  of 
national  assets,  which  are  not  represented  in  this  study  due  to  the  security  issues 
involved  but  which  would  undoubtedly  reflect  interesting  trends  in  discrepancy  data. 
In  general,  this  issue  can  be  mitigated  using  the  data  masking  and  normalizing 
techniques  explained  in  Chapter  5. 

•  Exposure  of  weaknesses.  There  is  often  a  general  uneasiness,  if  not  fear,  amongst 
operators  about  releasing  discrepancy  data.  It  exposes  intimate  details  about  the 
nature  of  an  operations  facility  that  most  people  would  prefer  remain  private.  Some 
managers  and  supervisors  may  feel  that  discrepancies  reflect  poorly  on  their  ability  to 
run  an  operations  facility,  while  rank-and-file  operators  may  feel  discrepancies 
indicate  a  lack  of  skill.  Unfortunately,  this  hesitation  is  a  result  of  human  nature  and 
is  difficult  to  overcome.  One  solution  is  to  reassure  operators  of  the  non-attribution 
method  of  data  collection,  masking,  and  reporting  by  an  impartial  third  party,  like 
LAI. 

•  Additional  work/cost.  Publishing  data  -  discrepancy  or  otherwise  -  does  in  fact 
require  an  additional  measure  of  labor  on  the  part  of  the  operations  staff.  In  many 
cases,  cost  and  staffing  constraints  may  be  so  tight  that  the  organization  cannot  afford 
the  additional  work.  This  problem  can  be  overcome  if  an  externally  funded  third 
party  is  available  to  perform  the  analysis  at  no  additional  cost  and  minimal  effort  on 
the  part  of  the  operations  staff.  This  study  was  designed  to  minimize  the  impact  of 
participation  to  approximately  10  labor-hours  per  organization,  which  typically 
included  a  background  briefing,  the  preparation  and  submission  of  data,  a  follow-up 
interview,  and  a  post-study  debriefing. 
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•  Lack  of  perceived  value.  The  benefits  to  an  operations  organization  of  publishing  its 
discrepancy  data  are  not  immediately  apparent.  Since  so  few  organizations  currently 
publish,  or  even  analyze,  their  discrepancy  data,  it  is  easy  to  question  the  utility  of 
such  a  practice.  One  of  the  goals  of  this  study  is  to  make  the  case  for  discrepancy 
data  research  by  presenting  the  type  of  useful  information  gleaned  from  a  preliminary 
analysis.  The  applications  of  discrepancy  data  research  are  explored  more  fully  later 
in  this  chapter,  and  results  of  the  analysis  are  presented  in  Chapter  6. 

•  Lack  of  precedent.  When  a  particular  practice  is  not  commonplace  within  the 
aerospace  community,  as  is  the  case  with  discrepancy  data  analysis,  it  typically  takes 
a  determined  effort  to  overcome  institutional  inertia.  The  effort  involved  in  changing 
the  course  of  inertia  and  adopting  a  new  practice  may  be  too  great  even  for 
proponents  of  the  practice.  At  the  very  least,  this  paper  can  serve  as  one  reference 
that  discrepancy  data  analysis  is  indeed  possible,  and  that  even  a  rudimentary  analysis 
can  reveal  meaningful  insights  into  the  conduct  of  satellite  operations. 

The  methods  used  for  this  study  were  designed,  in  part,  to  address  the  concerns  highlighted 
above.  Sponsorship  by  the  Lean  Aerospace  Initiative  was  crucial  for  maintaining  an 
unbiased,  impartial  third-party  perspective  when  dealing  with  several  different  organizations, 
some  of  which  are  in  direct  competition  with  each  other. 


Impacts  of  the  Lack  of  Discrepancy  Data 

Since  there  is  no  basis  for  comparison,  the  impacts  of  the  lack  of  detailed  discrepancy  data 
cannot  be  assessed  in  a  quantitative  fashion.  However,  this  section  will  point  out  areas  in 
which  existing  problems  within  the  operations  community  can  be  addressed  in  part  using  a 
thorough  analysis  of  discrepancy  data. 

Recent  government  acquisition  reform  efforts  advocate,  among  other  things,  an  increased 
awareness  of  system  operability.  Emphasis  is  placed  on  acquiring  systems  that  are  designed 
to  improve  operability.  But  what  exactly  constitutes  ‘operability’  for  a  space  system? 
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According  to  the  U.S.  Air  Force’s  2025  Final  Report,  the  definition  is  “the  ability  to  mount 
and  sustain  aerospace  operations  regardless  of  the  nature  of  threat,  level  of  conflict, 
environmental  conditions,  and/or  geographic  location.”9  But  more  precise  definitions  of 
operability  for  a  given  space  system  can  vary  depending  on  its  mission  and  the  individual 
operators  involved. 

Since  there  is  no  universal  definition  for  operability,  another  approach  is  to  examine  what 
operability  is  not.  Space  systems  that  are  prone  to  problems  are  less  ‘operable’  than  space 
systems  that  are  consistently  trouble-free.  Thus,  one  way  to  at  least  partially  improve 
operability  for  a  given  space  system  is  to  find  ways  to  prevent  problems  from  occurring. 

With  a  lack  of  operational  discrepancy  data,  however,  design  engineers  revert  back  to  what 
they  know  -  how  to  prevent  problems  onboard  the  spacecraft  -  when  in  fact,  they  might  be 
overlooking  several  larger  sources  of  problems  in  the  process.  An  analysis  of  discrepancy 
data  from  similar  space  systems  would  identify  the  largest  potential  sources  of  problems  for  a 
new  space  system. 

There  are  a  few  examples  of  satellite  programs  that  use  a  subset  of  operational  discrepancy 
data  to  make  improvements  to  future  spacecraft,  including  the  Air  Force’s  Global  Positioning 
System  and  Defense  Support  Program  constellations,  as  well  as  NASA’s  TDRSS 
constellation.  This  practice  results  in  slight  modifications  to  each  spacecraft  leaving  the 
factory,  and  is  helpful  for  improving  the  overall  quality  of  the  product.  One  glaring 
similarity  is  that  all  three  programs  are  large,  multi-year  government  contracts,  which  makes 
it  relatively  easy  for  the  contractor  to  close  the  design-build-operate  feedback  loop  within  the 
company.  However,  companies  that  build  small  lots  of  spacecraft  (i.e.  one)  do  not  have  the 
same  luxury  and  would  require  historical  discrepancy  data  from  other  similar  programs  in 
order  to  have  the  same  effect. 


9  Mayes,  M.  Scott,  Felix  A.  Zambetti  III,  Stephen  G.  Harris,  Linda  K.  Fronczak,  and  Samuel  J.  McCraw. 
“Aerospace  Sanctuary  in  2025:  Shrinking  the  Bull’s-Eye.”  2025.  Final  Report,  vol.  2,  ch.  8.  Maxwell  AFB: 
Air  University,  1996.  p.  2. 
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Applications  for  Discrepancy  Data 

There  are  several  different  uses  for  discrepancy  data,  depending  on  the  role  of  the  individual 
performing  the  analysis.  This  section  summarizes  how  various  groups  can  use  discrepancy 
data  to  enhance  the  quality  of  their  work. 

•  System  architect.  Trade  studies  and  architectural  decisions  made  early  in  the 
concept  exploration  phase  have  a  tremendous  impact  on  the  final  implementation  of  a 
system.  One  way  to  avoid  overlooking  acceptable  solutions  is  to  create  an  array  of 
models  that  allow  an  automated  search  of  large  architectural  trade  spaces.  The  Space 
Systems,  Policy,  and  Architecture  Research  Consortium  (SSPARC)  methodology  is 
an  example  of  this  approach.  In  order  to  yield  meaningful  results,  however,  the 
models  used  for  the  analysis  must  contain  accurate  relationships  between  the  design 
inputs  and  the  predicted  performance  of  the  resulting  architecture.  Although  these 
relationships  are  generally  well  understood  for  the  performance  of  spacecraft 
hardware,  the  effect  design  decisions  have  on  the  performance  of  lifecycle  operations 
is  often  overlooked  or  ignored.  Historical  discrepancy  data  can  help  the  system 
architect  develop  operations  models  using  accurate,  realistic  relationships. 

•  Spacecraft  designer.  Engineers  occasionally  face  choices  on  how  to  implement  a 
particular  design  specification  in  a  spacecraft.  A  typical  trade  study  will  include 
factors  like  power  requirements,  weight  restrictions,  thermal  characteristics,  etc. 
Current  best  practices  also  include  factoring  the  reliability  of  the  component  into  the 
trade  study,  and  perhaps  consulting  with  an  operations  representative  to  get  a  feel  for 
which  solution  would  be  preferred.  Discrepancy  data  would  also  allow  the  engineer 
to  consider  the  impact  of  his  or  her  decision  on  the  operability  of  the  spacecraft  as 
another  variable  in  the  trade  study,  as  proposed  by  Brandon  Wood  in  “Development 
of  an  Operability  Metric  for  Analysis  of  Conceptual  Space  Systems.”10 

•  Operations  manager.  Operations  managers  can  use  the  methodology  outlined  in 
Chapter  5  to  analyze  their  own  organization’s  discrepancy  data.  The  analysis  will 

10  Wood,  Brandon  and  Joyce  Warmkessel.  “Development  of  an  Operability  Metric  for  Analysis  of  Conceptual 
Space  Systems.”  Cambridge,  MA:  Massachusetts  Institute  of  Technology,  2001. 
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help  identify  recurring  problems  as  areas  for  improvement.  When  compared  with 
discrepancy  data  from  other  organizations,  managers  can  benchmark  their 
performance  and  determine  whether  better  solutions  to  their  problems  are  available  in 
the  community  at  large. 

•  Program  manager.  In  the  most  general  sense,  discrepancy  data  will  help  program 
managers  understand  the  effect  of  discrepancies  on  the  lifecycle  cost  of  the  space 
system.  In  situations  where  they  are  presented  with  alternate  implementations  for  a 
new  system  or  proposed  changes  to  an  existing  system,  program  managers  can  use 
cost  figures  and  mission  impact  reports  from  existing  discrepancy  data  to  perform  a 
thorough  cost-benefit  analysis  and  make  a  well-informed  decision. 

The  most  noteworthy  point  regarding  these  applications  is  that,  in  almost  all  cases,  detailed 
discrepancy  data  already  exists.  Most  organizations  do  a  thorough  job  documenting 
problems  as  they  occur.  Unfortunately,  the  logs  are  typically  used  only  for  short-term 
problem  tracking  and  resolution.  Occasionally,  they  are  also  researched  within  the 
organization  and  used  as  a  reference  when  similar  problems  occur  later  in  the  operational 
lifetime.  Thus,  the  hard  work  of  recording  the  data  is  already  being  accomplished  -  all  that 
remains  is  to  make  use  of  it. 
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Chapter  3  -  Research  Overview  and  Hypotheses 


The  purpose  of  this  research  is  to  characterize  the  number  and  type  of  problems  satellite 
operators  encounter  during  the  course  of  routine  daily  activities.  This  chapter  surveys 
existing  literature  on  similar  topics,  provides  an  overview  of  the  research  strategy,  and 
presents  the  specific  hypotheses  addressed  by  the  study. 


Existing  Literature 

Past  research  in  the  area  of  operational  problems  focuses  almost  exclusively  on  hardware 
failures  onboard  the  spacecraft.  One  of  the  most  comprehensive  analyses  available  to  the 
general  public  is  Herbert  Hecht’s  technical  report,  “Reliability  Prediction  for  Spacecraft.”1 1 
Hecht’s  study  compiled  anomaly  reports  for  over  300  satellites  launched  between  the  early 
1960s  and  January  1984.  Spacecraft  anomalies  were  categorized  according  to  cause  and 
affected  subsystem,  and  the  subsequent  results  used  to  update  the  MIL-HDBK-217  model  for 
predicting  spacecraft  reliability. 

More  recent  anomaly  analyses  are  performed  by  the  Aerospace  Corporation’s  Space 
Operations  Support  Office  (SOPSO),  which  maintains  an  anomaly  database  with  historical 
data  for  over  400  satellites.  Operational  failure  modes  are  identified  based  on  cause, 
equipment  type,  and  subsystem.12  Correlation  studies  are  performed  to  determine  the  effect 
of  space  weather  and  other  factors  on  overall  spacecraft  performance. 


Some  studies  focus  on  spacecraft  anomalies  caused  by  specific  conditions  or  anomalies  that 
occur  in  special  circumstances.  One  example  is  Tosney  and  Boeck’s  compilation  of  satellite 
anomalies  caused  by  the  Leonid  meteor  shower.13  There  are  also  several  papers  that  examine 


11  Hecht,  Herbert,  and  Myron  Hecht.  “Reliability  Prediction  for  Spacecraft.”  Technical  report.  Griffiss  Air 
Force  Base,  NY:  Rome  Air  Development  Center.  December  1985. 

12  The  Aerospace  Corporation.  “SOPSO  Resources.”  Online  reference, 
http://www.aero.org/sopso/resources.html.  March  15,  2001. 

13  Tosney,  W.F.,  and  M.L.  Boeck.  “Integrated  satellite  anomaly  database  -  Overview,  community  access,  and 
plans  for  future  environemental  events.”  Leonid  II  Meteoroid  Storm  and  Satellite  Threat  Conference, 
Manhattan  Beach,  CA,  May  1 1-13,  1999,  Proceedings.  Los  Angeles,  CA:  Aerospace  Corporation,  1999. 
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various  hardware  anomalies  caused  by  the  radiation  environment.14  The  effects  of  many 
different  types  of  space-related  phenomena  have  been  studied  in  detail  and  documented. 

Such  studies  are  valuable  for  calculating  hardware  failure  rates,  reliability  specifications,  and 
model  calibration  constants,  but  do  not  capture  the  full  scope  of  problems  that  an  operator 
might  face.  On  a  routine  day-to-day  basis,  satellite  operators  may  encounter  problems 
completely  unrelated  to  the  spacecraft  and  unaffected  by  its  performance.  This  fact  prompted 
the  first  hypothesis  of  this  study,  described  later  in  this  chapter. 

Hypotheses 

Although  the  discrepancy  data  collected  for  this  study  can  help  individual  organizations 
answer  several  types  of  questions  particular  to  their  unique  environment,  the  scope  of  the 
study  is  intended  to  address  more  broadly  applicable  issues.  It  examines  and  tests  the 
following  statements: 

•  First  hypothesis.  Most  problems  encountered  by  an  operator  do  not  involve  the 
spacecraft  at  all,  but  are  attributed  to  other  elements  of  the  space  system. 

•  Second  hypothesis.  Correlations  exist  between  aspects  of  a  space  system  design  and 
the  nature  of  problems  experienced  by  the  operations  staff  over  the  long  term. 

The  first  hypothesis  investigates  the  notion  that  the  infrastructure  of  a  space  system  is 
generally  more  problem-prone  than  the  spacecraft  itself.  Although  there  is  a  measure  of 
agreement  within  the  operational  community  that  this  assertion  is  true,15  it  has  not  yet  been 
proved  or  disproved  in  an  analytical  fashion.  For  the  purposes  of  this  study,  the 
infrastructure  of  a  space  system  consists  of  the  following: 

•  Ground  system.  Includes  both  the  hardware  and  software  deployed  to  decrypt, 
decommutate,  calibrate,  display,  and  archive  satellite  telemetry  and  mission  data. 


14  Lauriente,  M.,  A.L.  Vampola,  R.  Koga,  and  R.  Hosken.  “Analysis  of  spacecraft  anomalies  due  to  the 
radiation  environment.”  Journal  of  Spacecraft  and  Rockets,  vol.  36,  no.  6,  Nov-Dec  1999,  pp.  902-906. 

15  Based  on  interviews  with  satellite  operators  from  several  different  organizations. 
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Also  includes  equipment  used  to  format,  validate,  and  encrypt  commands.  Most 
elements  of  the  human-computer  interface  fall  into  this  category. 

•  Communications  relay.  Includes  routers,  switches,  patch  panels,  landlines, 
microwave  transmissions,  satellite  relays,  ground  station  antennas  and  RF  equipment, 
and  communication  satellites.  The  exact  equipment  included  in  this  category  varies 
greatly  depending  on  the  ops  concept  used  for  a  particular  mission,  and  may  be 
anything  from  a  single,  dedicated  satellite  dish  to  a  shared,  global  communications 
network. 

•  Tracking  equipment.  Includes  hardware  and  software  necessary  to  perform  RF 
and/or  optical  pointing  and  ranging  techniques.  This  equipment  is  often  collocated 
with  elements  of  the  communications  relay,  but  may  also  be  provided  by  an  external 
source  as  a  service  to  the  program. 

•  Facilities.  Includes  the  physical  structures  in  which  equipment  and  personnel  are 
housed,  access  controls  to  said  structures,  utilities  such  as  water  and  electricity,  and 
heating  and  cooling  systems. 

•  Administrative  equipment.  Includes  all  other  hardware  and  software  used  to 
support  operations  either  directly  or  indirectly,  like  telephones,  fax  machines,  offline 
computers,  printers,  etc. 

The  second  hypothesis  investigates  the  possibility  that  certain  design  choices  typically  made 
early  in  the  product  development  phase  correlate  with  specific,  predictable  types  of  problems 
experienced  in  the  operational  phase.  Care  is  taken  here  not  to  imply  that  such  correlations 
indicate  cause-and-effect  relationships.  Since  the  true  root  cause  in  each  relationship  may 
vary  from  one  satellite  design  to  another,  root  cause  analyses  should  be  performed  on  a  case- 
by-case  basis  and  are  therefore  outside  the  scope  of  this  research.  However,  the  presence  of  a 
correlation  definitely  indicates  an  area  that  merits  further  investigation.  The  specific  design 
elements  examined  as  part  of  this  study  are: 
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•  Mission  type 

•  Communications  relay  scheme 

•  Spacecraft  altitude 

•  Orbit  characteristics 

•  Attitude  control  scheme 

•  Complexity 

•  Multiple  spacecraft/constellation 

Details  concerning  each  design  element  are  included  in  Chapter  4.  The  design  elements  were 
selected  on  the  basis  of  availability  of  pertinent  data  and  the  likelihood  of  a  correlation. 
Additional  design  elements  that  failed  to  meet  both  of  the  criteria,  but  are  still  valid  for  future 
study,  include: 

•  Available  reserve  on-board  memory  capacity  at  launch 

•  Telemetry/command  format  used 

•  Type  of  ground  system  software  used 

•  Number  of  telemetry  measurands  sampled 

•  Communications  subsystem  link  margin  at  launch 

•  Thermal  subsystem  design  margin  at  launch 

•  Solar  array/battery  type 


Research  Structure 

This  research  stems  from  key  questions  facing  the  Test  &  Space  Operations  focus  team 
within  LAI: 

•  What  kinds  of  problems  are  operators  having? 

•  Can  we  do  anything  to  fix  existing  problems,  predict  new  ones,  and  ultimately 
prevent  them  from  occurring? 

Recognizing  that  anomalous  conditions  increase  the  variability  and  cost  of  performing 
satellite  operations,  it  is  desirable  to  avoid  or  prevent  problems  wherever  possible.  It  is  not 
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coincidental  that  these  two  questions  directly  relate  to  the  lean  principles  of  eliminating 
waste  and  pursuing  perfection.  The  rest  of  this  section  provides  an  overview  of  how  these 
two  questions  were  translated  into  the  analysis  method  described  in  Chapter  5. 

Although  it  may  be  possible  to  address  the  first  key  question  within  a  strictly  theoretical 
framework,  such  an  approach  can  easily  become  divorced  from  reality.  Therefore,  the  best 
way  to  answer  the  question  is  to  characterize  the  types  of  actual  problems  satellite  operators 
experience  over  the  lifetime  of  each  vehicle  they  control.  Normally,  an  applied  approach  like 
this  would  require  researchers  to  implement  data  collection  procedures  at  each  facility 
targeted  for  study. 

Fortunately,  it  is  and  has  been  fairly  common  in  the  satellite  operations  community  to  log 
problems,  or  symptoms  of  problems,  when  they  occur.  Thus,  the  most  labor-intensive,  long¬ 
term  work  has  already  been  performed.  What  remained  was  to  collect  the  factual  data  from  a 
fairly  large  sample  of  organizations.  This  proved  to  be  more  difficult  than  expected,  due  to 
the  reasons  discussed  in  Chapter  2. 

Since  there  is  no  industry  standard  format  for  logging  discrepancy  data,  it  was  also  necessary 
to  develop  procedures  to  merge  each  log  on  a  case-by-case  basis.  For  some  databases,  it  was 
merely  a  matter  of  selecting  appropriate  fields  for  inclusion.  For  others,  the  procedure 
involved  manually  reviewing  each  discrepancy  in  order  to  properly  categorize  it.  Once  the 
submitted  data  was  compiled  into  one  data  set,  a  thorough  analysis  could  be  performed  using 
well-established  statistical  techniques. 

The  second  key  question  is  more  difficult  to  attack.  Although  the  main  thrust  of  the  question 
is  to  fix  existing  problems  and  eventually  prevent  new  problems  from  occurring,  such  an 
ambitious  goal  is  beyond  the  scope  of  this  research.  The  focus  will  be  instead  to  find  an 
additional  tool  for  predicting  problems  before  they  occur.  In  a  risk-adverse  community  such 
as  the  aerospace  sector,  this  goal  alone  would  be  helpful  if  accomplished.  And  once  the 
problems  are  identified,  managers,  engineers,  and  operators  have  considerable  knowledge 
and  expertise  in  dealing  with  them. 
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One  way  to  predict  future  problems  is  by  investigating  trends  in  the  discrepancy  reports  from 
similar  spacecraft  that  already  on-orbit.  Since  a  comprehensive  data  set  of  operational 
discrepancies  was  already  required  to  address  the  first  key  question,  what  remained  was 
finding  relationships  between  the  recorded  problems  and  an  unknown  set  of  independent 
variables.  This  research  had  to  identify  one  of  the  true  independent  variables  and  then 
determine  its  relationship  to  the  dependent  variables,  the  number  and  type  of  problems  that 
occur  during  lifecycle  operations. 

The  method  used  to  accomplish  this  task  was  a  simple,  brute  force  approach.  In  other  words, 
a  candidate  independent  variable  was  selected,  and  then  analyzed  to  determine  if  a 
relationship  existed  with  the  dependent  variables.  If  no  correlation  existed,  the  candidate 
independent  variable  was  discarded.  If  a  correlation  did  exist,  a  ‘true’  independent  variable 
was  considered  found.  Since  the  number  of  possible  independent  variables  is  virtually 
unlimited,  the  pool  from  which  candidate  variables  were  chosen  was  narrowed  down  using 
three  criteria: 

•  Design  element.  To  begin  with,  only  elements  of  a  typical  spacecraft  design  were 
considered  for  analysis  in  this  study. 

•  Potential  for  correlation.  The  candidate  independent  variables  selected  for 
examination  were  those  for  which  intuition  suggested  a  possible  relationship. 

•  Availability  of  data.  Of  the  remaining  variables,  the  ones  finally  studied  were  those 
for  which  sufficient  data  existed  to  perform  the  analysis. 

The  convergence  of  the  three  criteria  is  reflected  in  the  second  hypothesis,  below.  The 
approach  just  described  does  not  predict  a  problem  per  se,  but  identifies  the  potential  for  a 
problem  to  occur.  The  shortcomings  to  this  approach  -  namely,  predicting  future  problems 
based  on  past  performance,  merging  data  collected  using  different  procedures,  and  the 
existence  of  other  untested  independent  variables  -  will  be  discussed  in  more  detail  in 
Chapter  7. 
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Chapter  4  -  Discrepancies  Defined 

This  paper  defines  a  discrepancy  as  the  perception  by  an  operator  that  some  portion  of  the 
space  system  has  failed  to  operate  as  designed.  Thus,  discrepancies  can  include  ground 
system  malfunctions,  procedural  errors,  and  even  misdiagnosis  that  a  problem  exists  -  in 
addition  to  actual  spacecraft  anomalies.  This  chapter  further  defines  discrepancies  by 
describing  the  information  collected  for  each  one,  outlining  their  possible  causes,  and 
describing  the  affected  subsystems. 

Data  Collected  for  Each  Discrepancy 

As  noted  previously,  operators  log  discrepancy  events  each  time  they  occur.  Since  there  is 
no  standardized  format,  the  information  included  in  each  log  varies  from  one  organization  to 
another.  Procedures  at  one  facility  may  call  for  a  very  detailed  report  of  the  circumstances 
surrounding  the  incident  and  the  resulting  action  taken,  while  procedures  at  another  facility 
may  require  only  a  brief  description  of  the  symptoms  observed.  In  addition,  the  logs  vary 
considerably  based  on  the  style  of  the  operator  actually  recording  the  data. 

For  the  purpose  of  this  study,  however,  a  minimum  standard  discrepancy  report  had  to  be 
defined.  Due  to  level  of  variation  in  the  source  data,  only  the  most  basic  information  was 
included  in  the  format  to  avoid  excluding  substantial  blocks  of  source  data.  The  fields 
chosen  for  the  discrepancy  format  were: 

•  Organization.  The  pseudonym  used  to  represent  the  organization  responsible  for 
operating  the  spacecraft. 

•  Satellite  associated  with  discrepancy.  The  pseudonym  used  to  represent  the 
specific  satellite  associated  with  the  discrepancy.  In  some  cases,  discrepancies  were 
not  associated  with  any  one  particular  spacecraft  and  were  noted  as  such. 

•  Date  of  discrepancy.  The  date  on  which  the  discrepancy  occurred. 
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•  Description  of  discrepancy.  A  brief  summary  of  the  discrepancy.  Any  identifying 
information  was  replaced  with  a  generic  designator. 

•  Affected  subsystem.  The  subsystem  within  the  space  system  most  closely 
associated  with  the  discrepancy.  The  categorization  system  used  for  the  field  is 
described  later  in  this  chapter. 

•  Root  cause  (if  identified).  When  available  in  the  source  data,  the  root  cause  of  the 
discrepancy  was  included  in  the  report.  Discrepancies  for  which  the  root  cause  was 
not  identified  were  not  included  in  any  analysis  requiring  the  information.  The 
categorization  system  used  for  the  field  is  described  later  in  this  chapter. 

In  addition  to  the  information  collected  for  each  discrepancy,  several  data  points  pertaining  to 
design  elements  were  collected  for  each  spacecraft.  The  data  points  were  then  linked  to  all 
the  discrepancies  associated  with  the  corresponding  spacecraft.  The  design  elements 
consisted  of  the  following: 

•  Mission  type.  The  category  that  best  describes  the  mission  of  the  spacecraft. 

Choices  include  remote  sensing,  communications,  research  &  development,  weather, 
observatory,  and  other. 

•  Communications  relay  scheme.  Examines  the  primary  method  used  to  relay 
spacecraft  commands,  state  of  health  telemetry,  and  payload  data  from  the  mission 
control  center  to  the  spacecraft  and  back.  Choices  include  one  dedicated  ground 
station;  multiple  dedicated  ground  stations;  shared  ground  station  network, 
specifically  the  NASA  Deep  Space  Network,  the  Air  Force  Satellite  Control  Network, 
and  the  NASA  GN;  other  shared  ground  station  network;  and  shared  space  relay 
network,  specifically  NASA  TDRSS. 

•  Spacecraft  altitude.  Examines  the  altitude  region  in  which  the  satellite  operates. 
Choices  include  low,  for  orbits  located  entirely  in  the  region  below  1,000  km 
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altitude16;  medium,  for  orbits  located  entirely  in  the  region  1,000  -  25,000  km 
altitude;  high,  for  orbits  located  entirely  in  the  region  above  25,000  km  altitude;  and 
multiple,  for  elliptical  orbits  passing  through  two  or  more  of  the  previous  regions.17 

•  Orbit  characteristics.  Examines  a  subset  of  special  cases  for  the  type  of  orbit  used. 
Choices  include  sun-synchronous;  polar,  for  polar  and  near-polar  orbits  with  an 
inclination  in  the  range  83°  -  97°  (other  than  sun-synchronous);  geosynchronous; 
geosynchronous  transfer  orbit;  and  South  Atlantic  Anomaly,  for  orbits  which  pass 
through  the  SAA.  Note  that  these  categories  do  not  include  all  possible  orbit 
configurations.  These  special  cases  were  singled  out  due  to  the  potential  for  specific 
types  of  discrepancies  associated  with  each  one. 

•  Attitude  control  scheme.  Examines  the  type  of  attitude  control  incorporated  in  the 
spacecraft  design.  Choices  include  gravity  gradient  stabilized,  spin  stabilized,  three- 
axis  stabilized,  and  free  tumble. 

•  Complexity.  This  general  qualitative  description  is  a  composite  of  several  design 
elements,  including  number  of  telemetry  measurands  sampled,  number  of  nodes  in 
the  communications  relay,  number  of  payloads  on  board  the  spacecraft,  acceptable 
tolerances  for  operating  constraints  (e.g.  pointing  accuracy,  stationkeeping  window, 
and  thermal  limits),  complexity  of  onboard  software,  and  time  criticality  for 
performing  operational  activities.  Choices  include  low,  medium,  and  high. 

•  Multiple  spacecraft/constellation.  Examines  whether  the  satellite  is  part  of  a 
production  line  of  multiple  similar  spacecraft,  and  if  so,  whether  the  spacecraft  is  the 
first  off  the  line.  Choices  include  no,  first,  and  yes. 


16  Gomey,  D.J.,  J.B.  Blake,  H.C.  Koons,  M.  Schulz,  A.L.  Vampola,  R.L.  Walterscheid,  and  J.R.  Wertz.  “The 
Space  Environment  and  Survivability.”  Space  Mission  Analysis  and  Design.  2nd  ed.  Torrance,  CA: 
Microcosm,  1993.  pp.  199-201. 

17  The  given  regions  are  selected  to  correspond  with  the  approximate  floor  altitude  of  the  inner  and  outer  zones 
of  the  Van  Allen  radiation  belts,  which  provide  convenient  demarcation  points. 
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Lastly,  each  organization  included  in  the  data  set  was  categorized  as  one  of  the  following 
types:  civil,  military,  commercial,  academic,  and  other.  The  data  for  organization  type  was 
then  linked  to  all  the  discrepancies  reported  by  the  corresponding  organization.  The 
categorization  system  described  above  provided  a  concise  way  to  group  similar  types  of 
discrepancies  during  the  analysis  process. 


Causes  of  Discrepancies 

Discrepancies  can  be  caused  by  a  number  of  factors.  When  a  root  cause  was  identified  in  the 
source  data,  one  of  the  following  categories  was  assigned  to  the  corresponding  discrepancy:18 

•  Employee/operator.  Discrepancies  caused  by  a  person  incorrectly  executing  a 
procedure,  sending  an  unintended  command,  altering  mission  critical  equipment,  etc. 

•  Design.  Discrepancies  caused  by  problems  in  the  design  of  the  spacecraft  or  ground 
system.  Applied  in  cases  where  the  component  in  question  is  verified  as  meeting 
design  specifications  but  still  causing  an  undesirable  condition. 

•  Procedure.  Discrepancies  caused  when  a  procedure  is  executed  as  written  and  later 
identified  as  incorrectly  planned.  Weigel’s  paper  originally  included  this  category  as 
part  of  Design,  but  is  separated  here  due  to  the  procedural  nature  of  operations. 

•  Material.  Discrepancies  caused  by  defective  equipment,  parts,  material,  etc.  on  the 
spacecraft,  as  can  best  be  determined  through  analysis.  Also  includes  equipment  or 
components  on  the  spacecraft  that  have  failed  to  meet  design  specifications. 

•  Equipment.  Discrepancies  caused  by  defective  equipment,  communications  lines, 
computers,  cables,  etc.  not  on  the  spacecraft.  This  category  focuses  primarily  on 
instances  of  verified  hardware  failure. 


18  Weigel,  A.L.  “Spacecraft  System-Level  Integration  and  Test  Discrepancies:  Characterizing  Distributions  and 
Costs.”  Cambridge,  MA:  Massachusetts  Institute  of  Technology,  2000.  pp.  25-26. 
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•  Software.  Discrepancies  caused  by  software,  either  on  the  spacecraft  or  on  the 
ground  equipment.  Includes  problems  caused  by  hung  processes,  or  instances  where 
a  computer  reboot  is  required  to  restore  functionality. 

•  Interference.  Discrepancies  caused  by  radio  frequency  interference  with  other 
spacecraft,  communications  relay  line  quality  degradation  or  noise,  or  scheduling 
conflicts  with  other  spacecraft. 

•  Weather.  Discrepancies  caused  by  external  environmental  factors,  including  storms 
that  affect  ground  stations,  single  event  upsets  on  the  spacecraft  caused  by  radiation, 
problems  associated  with  suspected  debris  or  meteor  hits,  etc. 

•  No  Anomaly.  Discrepancies  written  up  in  error,  or  determined  later  to  not  be 
anomalies,  etc.  This  includes  discrepancy  reports  written  up  for  informational 
purposes,  or  to  request  adjustments  to  telemetry  limit  values. 

•  Unknown.  Discrepancies  whose  cause  is  unknown  or  unable  to  be  determined. 

•  Other.  Discrepancies  that  do  not  fall  into  the  previous  eight  categories. 

In  general,  discrepancies  have  one  root  cause.  In  the  occasional  case  that  a  discrepancy  had 
two  or  more  root  causes,  it  counted  against  the  multiple  corresponding  categories  above. 

Affected  Subsystems 

Discrepancies  are  usually  associated  with  one  particular  subsystem  or  another.  The 
following  categories  were  used  for  subsystems  onboard  the  spacecraft : 

•  Electrical  Power  and  Distribution  System  (EPDS).  EPDS’s  primary  function 
includes  the  generation,  regulation,  storage,  and  distribution  of  electrical/electronic 
power  throughout  the  vehicle.  Other  names:  Electrical  Power  Subsystem  (EPS), 
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Power  Subsystem,  Power. 19  Includes,  but  is  not  limited  to,  solar  arrays,  batteries, 
switching  circuitry,  and  power  supply  regulators. 

•  Guidance,  Navigation  and  Control  (GNC).  The  GNC  subsystem’s  primary 
function  provides  determination  of  orbit  and  attitude,  plus  pointing  of  spacecraft  and 
appendages.  Other  names:  Attitude  Control  Subsystem  (ACS),  Attitude 
Determination  and  Control  Subsystem  (ADCS).20  Includes,  but  is  not  limited  to,  sun 
sensors,  horizon  sensors,  gyros,  magnetic  torquers,  GPS  receiving  equipment,  and 
attitude  determination  and  control  processors. 

•  Payload.  The  payload  subsystem’s  primary  function  provides  mission  specific 

capabilities  to  the  space  vehicle’s  functionality.  This  is  the  single  most  significant 

driver  of  spacecraft  design.  Payloads  have  various  capabilities  such  as 

21 

communication,  navigation,  science,  imaging,  radar,  and  others. 

•  Propulsion  (Prop).  The  propulsion  subsystem’s  primary  function  provides  thrust  to 
adjust  orbit  and  attitude,  and  to  manage  angular  momentum.  Other  names:  Reaction 
Control  Subsystem  (RCS).22  Includes,  but  is  not  limited  to,  propellant  tanks, 
thrusters,  plumbing,  and  valves. 

•  Structures  and  Mechanisms  Subsystem  (SMS).  SMS’s  primary  function  provides 
support  structure,  booster  adaptation,  and  moving  parts.  Other  names:  Structural, 
Structures  and  Mechanisms.23  Includes,  but  is  not  limited  to,  trusses,  panels,  hinges, 
and  pyrotechnics. 


19  Quintero,  A.H.  “Space  Vehicle  Anomaly  Reporting  System  (SVARS)  Electronic  Data  Interchange  (EDI) 
Template.”  Los  Angeles,  CA:  Aerospace  Corporation,  1996.  p.  26. 

20  Quintero,  A.H.  “Space  Vehicle  Anomaly  Reporting  System  (SVARS)  Electronic  Data  Interchange  (EDI) 
Template.”  Los  Angeles,  CA:  Aerospace  Corporation,  1996.  p.  26. 

21  Quintero,  A.H.  “Space  Vehicle  Anomaly  Reporting  System  (SVARS)  Electronic  Data  Interchange  (EDI) 
Template.”  Los  Angeles,  CA:  Aerospace  Corporation,  1996.  p.  26. 

22  Quintero,  A.H.  “Space  Vehicle  Anomaly  Reporting  System  (SVARS)  Electronic  Data  Interchange  (EDI) 
Template.”  Los  Angeles,  CA:  Aerospace  Corporation,  1996.  p.  26. 

23  Quintero,  A.H.  “Space  Vehicle  Anomaly  Reporting  System  (SVARS)  Electronic  Data  Interchange  (EDI) 
Template.”  Los  Angeles,  CA:  Aerospace  Corporation,  1996.  p.  26. 
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•  Data  Management  Subsystem  (DMS).  The  Data  Management  Subsystem’s  primary 
function  distributes  commands  and  accumulates,  stores,  and  formats  data  from  the 
spacecraft  and  payload.  Other  names:  Command  and  Data  Handling  (C&DH), 
Spacecraft  Computer  System,  Spacecraft  Processor.24  Includes,  but  is  not  limited  to, 
telemetry  sampling  circuitry,  solid  state  memory,  data  recorders,  and  central 
processors. 

•  Telemetry,  Tracking,  and  Command  (TT&C).  The  TT&C  subsystem’s  primary 
function  provides  communication  with  ground  and  other  spacecraft.  Uplink  data 
consists  of  commands  and  ranging  tones  while  downlink  data  consists  of  status 
telemetry,  ranging  tones,  and  may  include  payload  data.  Other  names: 
Communications  Subsystem.  Includes,  but  is  not  limited  to,  receivers,  transmitters, 

7  c 

and  wide-angle  antennas. 

•  Thermal.  The  Thermal  Control  Subsystem’s  primary  function  maintains  spacecraft 
equipment  within  allowed  temperature  range.  Other  names:  TCS,  Environmental 
Control  Subsystem  (ECS).  Includes,  but  is  not  limited  to,  radiators,  louvers,  heat 
sinks,  heaters,  and  cryogenic  cooling  systems.26 

•  Wiring  and  Cabling  (Harness).  Wiring  and  cabling  that  is  not  considered  part  of  a 
particular  subsystem  called  out  above.27 

•  Other  Spacecraft.  Discrepancies  that  are  traceable  down  to  the  subsystem  level,  but 
the  subsystem  does  not  fall  into  one  of  the  above  categories. 


24  Quintero,  A.H.  “Space  Vehicle  Anomaly  Reporting  System  (SVARS)  Electronic  Data  Interchange  (EDI) 
Template.”  Los  Angeles,  CA:  Aerospace  Corporation,  1996.  p.  26. 

25  Quintero,  A.H.  “Space  Vehicle  Anomaly  Reporting  System  (SVARS)  Electronic  Data  Interchange  (EDI) 
Template.”  Los  Angeles,  CA:  Aerospace  Corporation,  1996.  p.  26. 

26  Quintero,  A.H.  “Space  Vehicle  Anomaly  Reporting  System  (SVARS)  Electronic  Data  Interchange  (EDI) 
Template.”  Los  Angeles,  CA:  Aerospace  Corporation,  1996.  p.  26. 

27  Weigel,  A.L.  “Spacecraft  System-Level  Integration  and  Test  Discrepancies:  Characterizing  Distributions  and 
Costs.”  Cambridge,  MA:  Massachusetts  Institute  of  Technology,  2000.  p.  25. 

28  Weigel,  A.L.  “Spacecraft  System-Level  Integration  and  Test  Discrepancies:  Characterizing  Distributions  and 
Costs.”  Cambridge,  MA:  Massachusetts  Institute  of  Technology,  2000.  p.  25. 
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•  Spacecraft.  Discrepancies  that  cannot  be  traced  down  to  a  particular  subsystem 
called  out  above  fall  into  this  category.29 

The  following  categories  were  used  for  subsystems  other  than  those  on  the  spacecraft : 

•  Ground  System.  Includes  both  the  hardware  and  software  deployed  to  decrypt, 
decommutate,  calibrate,  display,  and  archive  satellite  telemetry  and  mission  data. 

Also  includes  hardware  and  software  used  to  format,  validate,  and  encrypt 
commands.  Most  elements  of  the  human-computer  interface  fall  into  this  category. 
This  equipment  is  typically  all  located  in  the  mission  operations  control  center. 

•  Communications  Relay.  Includes  routers,  switches,  patch  panels,  landlines, 
microwave  transmissions,  satellite  relays,  ground  station  antennas  and  RF  equipment, 
and  communication  satellites.  The  exact  equipment  included  in  this  category  varies 
greatly  depending  on  the  ops  concept  used  for  a  particular  mission,  and  may  be 
anything  from  a  single,  dedicated  satellite  dish  to  a  shared,  global  communications 
network. 

•  Tracking  Equipment.  Includes  hardware  and  software  necessary  to  perform  RF 
and/or  optical  pointing  and  ranging  techniques.  This  equipment  is  often  collocated 
with  elements  of  the  communications  relay,  but  may  also  be  provided  by  an  external 
source  as  a  service  to  the  program. 

•  Facilities.  Includes  the  physical  structures  in  which  equipment  and  personnel  are 
housed,  access  controls  to  said  structures,  utilities  such  as  water  and  electricity,  and 
heating  and  cooling  systems. 

•  Administrative  Equipment.  Includes  all  other  hardware  and  software  used  to 
support  operations  either  directly  or  indirectly,  like  telephones,  fax  machines,  offline 
computers,  printers,  etc. 

29  Weigel,  A.L.  “Spacecraft  System-Level  Integration  and  Test  Discrepancies:  Characterizing  Distributions  and 

Costs.”  Cambridge,  MA:  Massachusetts  Institute  of  Technology,  2000.  p.  25. 
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•  Other  Infrastructure.  Includes  subsystems  not  belonging  to  any  of  the  five 
categories  listed  above. 

In  general,  discrepancies  are  associated  with  one  particular  subsystem.  In  the  occasional  case 
that  a  discrepancy  affected  two  or  more  subsystems,  it  counted  against  the  multiple 
corresponding  categories  above. 
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Chapter  5  -  Methods  and  Procedures 

This  chapter  provides  a  detailed  explanation  of  the  procedure  used  to  collect,  convert,  and 
merge  source  discrepancy  data  from  each  contributing  organization.  Background 
information  on  the  statistical  methods  used  to  analyze  the  compiled  data  set  is  presented, 
along  with  a  description  of  the  steps  involved  in  the  actual  analysis. 


Data  Collection  Procedure 

The  first  step  in  collecting  source  discrepancy  data  was  to  contact  a  member  of  the  operations 
staff  from  each  program  of  interest,  either  directly  or  by  referral.  Generally,  the  individual 
authorized  to  release  source  data  -  and  therefore  the  best  to  contact  directly  -  was  the 
Mission  Director,  Mission  Operations  Manager,  or  equivalent.  A  background  briefing  that 
explained  the  study  and  its  objectives  was  provided  to  each  participating  organization.  After 
further  discussions  and  occasionally  legal  department  review,  the  source  data  was  released 
for  research  purposes. 

When  provided  in  the  form  of  hand- written  logs,  individual  entries  were  first  manually 
entered  into  electronic  format.  Hardcopy  forms  and  incident  reports  were  scanned  using 
optical  character  recognition  (OCR)  software  and  reviewed  for  accuracy.  Source  data 
provided  in  the  form  of  proprietary  or  mission-specific  electronic  databases  were  first 
exported  to  tab-delimited  text  files.  In  each  of  the  three  cases,  the  result  was  a  generic  text 
file  containing  all  the  original  source  data,  which  could  be  manipulated  by  a  variety  of 
software  tools. 

All  of  the  source  data  sets  used  a  unique  format  in  terms  of  structure,  data  requested,  field 
names,  etc.  In  order  merge  the  data  into  one  uniform  database,  several  modifications  were 
made  to  each  set.  To  begin  with,  a  standard  data  format  was  defined  using  a  core  set  of 
information  common  to  a  large  majority  of  the  source  sets: 
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•  Organization.  The  pseudonym  used  to  represent  the  organization  responsible  for 
operating  the  spacecraft. 

•  Satellite  associated  with  discrepancy.  The  pseudonym  used  to  represent  the 
specific  satellite  associated  with  the  discrepancy.  In  some  cases,  discrepancies  were 
not  associated  with  any  one  particular  spacecraft  and  were  noted  as  such. 

•  Date  of  discrepancy.  The  date  on  which  the  discrepancy  occurred. 

•  Title.  A  short  identifier  for  the  discrepancy  entry,  usually  a  phrase  or  single  sentence. 

•  Description.  A  brief  summary  of  the  discrepancy.  Any  identifying  information  was 
replaced  with  a  generic  designator. 

•  Affected  subsystem.  The  subsystem  within  the  space  system  most  closely 
associated  with  the  discrepancy.  The  categorization  system  used  for  the  field  is 
described  in  Chapter  4. 

•  Root  cause  (if  identified).  When  available  in  the  source  data,  the  root  cause  of  the 
discrepancy  was  included  in  the  report.  Discrepancies  for  which  the  root  cause  was 
not  identified  comprised  7%  of  all  reports  collected,  and  were  not  included  in  any 
analysis  requiring  that  information. 

Unnecessary  or  extraneous  data  fields  were  removed  from  the  source  data  sets,  and  the 
pseudonym  for  the  contributing  organization  was  added  to  every  entry  in  the  file.  In  a  few 
cases,  the  source  data  set  did  not  explicitly  break  out  one  of  the  fields  listed  above  (e.g. 
affected  subsystem),  and  the  data  for  that  field  had  to  be  manually  extracted  from  one  of  the 
other  fields,  usually  the  description.  Figure  5  shows  the  database  tool  used  to  review  and 
import  each  discrepancy  report.  Following  these  modifications,  the  data  sets  uniformly 
consisted  of  the  information  listed  above. 
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Figure  5  -  Discrepancy  Report  Import  Tool 


|  At  the  change  of  the  dap,  the  GMT  went  from  1 996  DAY  365  to  1 997 
|  DAY  001 .  Initially  on|y  a  few  areas  were  impacted  such  as  no  clock 
I  correlation  and  no  delogs.  At  02:54  however  FEP1  and  FEP2  clock 
|  lines  were  reinitialize  somehow.  The  new  GMT  then  became  70-001  - 
00:00:00  and  cEd  not  increment.  At  this  point  all  operations  dealing  with 
1  the  GMT  time  tag  failed.  This  included  such  things  as  inabifity  to  open 
|  history  and  block  files  and  inability  to  receive  UPDs.  It  should  be  noted 
1  that  the  GMT  clocks  were  fine  and  were  able  to  change  to  DAY  366. 


At  the  change  of  the  day.  the  GMT  went  from  1 996 
|  DAY  365  to  1 997  DAY  001 .  Initially  only  a  few  areas 
§:{  were  impacted  such  as  no  clock  correlation  and  no 
delogs.  At  02:54  however  FEP1  and  FEP2  clock  ines 
were  reinitialize  somehow.  The  new  GMT  then  became 
70-001  -00:00:00  and  did  not  increment  At  this  point  all 
t'  operations  dealing  with  the  GMT  lime  tag  failed.  This 
||l|j  included  such  things  as  inability  to  open  history  and 
:;lir  block  files  and  inability  to  receive  UPDs.  It  should  be 
:  noted  that  the  GMT  clocks  were  fine  and  were  able  to 

change  to  DAY  366. 


It  was  discovered  that  the  problem  was  software  related.  (Organization) 
maintenance  then  suggested  the  idea  to  manually  enter  the  right  time 
into  the  (Component)  time  decoder  card  on  the  front  end.  This  was 
done  by  running  a  cable  from  the  time  decoder  card  to  the  keyboard 
connected  to  the  FEP.  Before  the  time  was  inputted  the  reset  button 
was  pushed  on  the  front  end. 


■Illllllli 


Record:  »<  1  <  If 


I  Working  closely  with  (Organization),  who  was  experiencing  similar 
|  problems,  it  was  determined  that  maybe  it  was  a  (Organization)  fine 
problem  Notified  the  (Organization)  switch.  It  was  determined  that 
III  there  are  two  lines  feeding  off  the  (Organization)  (Component) 
generator.  These  two  lines  supply  the  GMT  time  for  all  of  the 
|  operations  centers.  One  was  working  correctly  and  one  did  not 
|  recognize  the  leap  year.  This  was  fixed  and  at  that  point  both  FEP1 
I  and  FEP2  could  not  recognize  the  GMT  time  at  all  and  could  only 
::  display  70-001  -00:00:00. 


|  Every  lime  the  front  end  reset  button  is  pushed  it  resets  the  GMT  card. 
•  so  if  the  front  end  needs  to  rebooted  (Organization)  maintenance  must 
|  be  called  again  to  input  the  correct  time  manually.  Operations  should 
?■  return  to  norma!  when  1997  comes. 


Accept 


The  remaining  data  preparation  steps  were  accomplished  to  address  concerns  about  the 
release  of  proprietary  data.  Any  occurrence  in  the  data  of  a  spacecraft  name  was  replaced 
with  the  corresponding  pseudonym.  References  to  payload  instruments,  personnel  teams, 
unique  facilities,  or  specific  equipment  configurations  were  replaced  with  generic  descriptive 
terms.  At  this  point  in  the  process,  each  data  set  was  purged  of  identifying  information  and 
ready  to  be  added  to  the  discrepancy  database. 


Statistics  Background 

It  is  important  to  note  that  the  true  population  of  interest  is  the  set  of  all  discrepancies 
occurring  for  all  spacecraft  launched  by  all  nations  since  the  first  satellite,  Sputnik.  The  only 
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caveat  is  that  recent  discrepancy  data  is  more  relevant,  and  therefore  of  more  interest,  than 
discrepancy  data  from  older  programs.  Unfortunately,  it  is  impossible  to  measure  the  entire 
population,  for  reasons  discussed  in  Chapter  7.  Therefore,  only  a  sample  of  the  population 
was  collected  and  measured  for  this  study.  Discrepancy  data  for  on-orbit  spacecraft  were 
requested  from  several  organizations  in  a  variety  of  mission  areas.  Strictly  speaking,  the 
emphasis  on  spacecraft  that  are  currently  operational  is  not  a  representative  sample  of  the 
entire  population.  It  does,  however,  capture  data  that  is  the  most  relevant  to  other  current  and 
future  programs. 

Key  characteristics  of  the  sample  set  are  used  to  draw  conclusions  about  the  greater 
population.  Although  this  approach  is  well  established  and  widely  accepted,  it  is  necessary 
to  verify  the  proper  application  of  the  technique  to  this  particular  case.  Since  they  form  the 
foundation  of  the  analysis  conducted  here,  the  concepts  associated  with  basic  inferential 
statistics  are  briefly  reviewed  next.  A  more  thorough  treatment  of  the  subject  can  be  found  in 
Mendenhall  and  Sincich’s  Statistics  for  Engineering  and  the  Sciences.30 

Measures  of  Central  Tendency 

Measures  of  central  tendency  are  the  first  of  four  groups  of  numerical  descriptive  measures 
computed  from  a  set  of  quantitative  data.  They  are  a  collection  of  mathematical  techniques 
used  to  help  locate  the  center  of  the  relative  frequency  distribution.  Each  technique  has 
strengths  and  weaknesses  that  make  it  more  useful  and  accurate  in  some  cases,  but  not  others. 
The  ‘best’  measure  of  central  tendency  for  a  given  data  set  depends  both  on  the  type  of 
descriptive  information  desired  and  the  nature  of  the  data  being  evaluated.  This  section 
reviews  all  of  the  measures  of  central  tendency  used  in  the  discrepancy  analysis. 

Sample  Mean 

The  sample  mean  of  a  set  of  n  measurements,  y\,yi,  ...,y3,  taken  from  a  larger  population  is 
the  average  of  the  measurements: 


30  Mendenhall,  William  and  Terry  Sincich.  Statistics  for  Engineering  and  the  Sciences.  3rd  ed.  San  Francisco, 
CA:  Dellen  Publishing  Company,  1992. 
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Although  the  mean  is  often  the  preferred  measure  of  central  tendency,  it  is  sensitive  to  very 
large  or  very  small  observations.  Consequently,  the  mean  will  shift  toward  the  direction  of 
skewness  and  may  be  misleading  in  some  situations. 

Sample  Median 

The  median  of  a  set  of  n  measurements,  yhy2, . . .,  yz,  is  the  middle  number  when  the 
measurements  are  arranged  in  ascending  (or  descending)  order.  In  other  words,  it  is  the 
value  ofy  located  so  that  half  the  area  under  the  relative  frequency  histogram  lies  to  its  left 
and  half  the  area  lies  to  its  right.  If  the  number  of  measurements  in  a  data  set  is  odd,  the 
median  is  the  measurement  that  falls  directly  in  the  middle  when  the  measurements  are 
arranged  in  ascending  order.  If  the  number  of  measurements  is  even,  the  median  is  defined 
as  the  average  of  the  two  middle  measurements  when  the  measurements  are  arranged  in 
ascending  order.  The  median  is  sometimes  called  a  resistant  measure  of  central  tendency 
since  it,  unlike  the  mean,  is  resistant  to  the  influence  of  extreme  observations.  For  data  sets 
that  are  extremely  skewed,  the  median  would  better  represent  the  center  of  the  distribution 
data.32 

5%  Trimmed  Mean 

The  5%  trimmed  mean  represents  a  compromise  between  the  two  extremes  of  sample  mean 
and  sample  median.  It  is  computed  by  ordering  the  values  within  the  data  set  from  smallest 
to  largest,  trimming  5%  of  the  values  from  the  top  and  5%  of  the  values  from  the  bottom  of 
the  data  set,  and  then  computing  the  usual  sample  mean  as  described  above  for  the  data  that 
remain.  This  prevents  unusual  outlier  and  extreme  values  in  the  tails  of  the  distribution  from 
affecting  the  size  of  the  sample  mean,  and  makes  the  trimmed  mean  measurement  more 
resistant  than  the  sample  mean  measurement.  While  it  still  is  not  as  resistant  as  the  sample 
median  measurement,  the  benefit  of  the  trimmed  mean  measurement  is  that  it  is  based  on 

31  Entire  paragraph  drawn  from  Mendenhall,  William  and  Terry  Sincich.  Statistics  for  Engineering  and  the 
Sciences.  3rd  ed.  San  Francisco,  CA:  Dellen  Publishing  Company,  1992.  pp.  28-29. 

32  Entire  paragraph  drawn  from  Mendenhall,  William  and  Terry  Sincich.  Statistics  for  Engineering  and  the 
Sciences.  3rd  ed.  San  Francisco,  CA:  Dellen  Publishing  Company,  1992.  pp.  28-29. 
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more  values  than  the  sample  median  measurement  while  effectively  dealing  with  outliers  and 

'X'X 

extreme  values.  The  number  of  data  points  to  be  excluded  from  each  tail  of  the  data  set  is 
calculated  by  multiplying  the  total  number  of  observations  by  0.05  and  rounding  the  result 
down  to  the  nearest  integer. 

Measures  of  Variation 

Measures  of  variation  are  the  second  group  of  numerical  descriptive  measures.  They  are  a 
collection  of  mathematical  techniques  used  to  describe  the  spread  of  measurements  over  the 
entire  range  of  values.  As  before,  each  technique  has  strengths  and  weaknesses  that  make  it 
more  useful  and  accurate  in  some  cases,  but  not  others.  The  ‘best’  measure  of  variation  for  a 
given  data  set  depends  both  on  the  type  of  descriptive  information  desired  and  the  nature  of 
the  data  being  evaluated.  This  section  reviews  all  of  the  measures  of  variation  used  in  the 
discrepancy  analysis. 

Sample  Range 

The  range  is  equal  to  the  difference  between  the  largest  measurement  and  the  smallest 
measurement  in  a  data  set.  It  is  possible  that  two  different  data  sets  could  possess  the  same 
range,  but  differ  greatly  in  the  amount  of  variation  in  the  data.  Consequently,  the  range  is  a 
relatively  insensitive  measure  of  data  variation.34 


Sample  Variance 

The  sample  variance  of  a  set  of  n  measurements,^,)^,  ...,>>3,  taken  from  a  larger  population 
is  defined  to  be: 


S2  =-*=!- 


S( yi-yf  Et,2 


f  n 

!>/ 

V 1-1  , 


1=1 


n- 1 


n- 1 


(2) 


33  Entire  paragraph  drawn  from  Wilcox,  Rand  R.  Statistics  for  the  Social  Sciences.  San  Diego:  Academic 
Press,  1996.  pp.  15-16. 

34  Entire  paragraph  drawn  from  Mendenhall,  William  and  Terry  Sincich.  Statistics  for  Engineering  and  the 
Sciences .  3rd  ed.  San  Francisco,  CA:  Dellen  Publishing  Company,  1992.  pp.  30-31. 
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The  sample  variance  is  primarily  of  theoretical  significance,  but  is  important  in  that  it  is  used 
to  derive  the  sample  standard  deviation.35 

Sample  Standard  Deviation 

The  standard  deviation  of  a  sample  of  n  measurements  is  equal  to  the  square  root  of  the 
variance: 

y)2 

-  (3) 

The  standard  deviation  as  a  measure  of  data  variation  is  easily  interpreted  by  means  of  a  rule 
of  thumb  known  as  the  Empirical  Rule. 

The  Empirical  Rule 

If  a  data  set  has  an  approximately  mound-shaped  relative  frequency  distribution,  then  the 
following  rules  of  thumb  may  be  used  to  describe  the  data  set: 

•  Approximately  68%  of  the  measurements  will  lie  within  1  standard  deviation  of  their 
mean  (i.e.  within  the  interval  y  ±  s  for  samples). 

•  Approximately  95%  of  the  measurements  will  lie  within  2  standard  deviations  of  their 
mean  (i.e.  within  the  interval  y±2s  for  samples). 

•  Almost  all  the  measurements  will  lie  within  3  standard  deviations  of  their  mean  (i.e. 
within  the  interval  y  ±  3s  for  samples). 

The  percentages  given  in  the  rule  are  only  approximate,  particularly  for  the  first  interval  of 
one  standard  deviation.  The  percentage  of  the  total  number  of  measurements  that  fill  within 
two  standard  deviations  of  their  mean  will  usually  be  quite  close  to  95%.  The  Empirical 
Rule  is  the  result  of  the  practical  experience  of  researchers  in  many  fields  who  have  observed 
its  validity  with  many  different  types  of  data  sets.37 


2.0',  - 


1=1 


n- 1 


35  Entire  paragraph  drawn  from  Mendenhall,  William  and  Terry  Sincich.  Statistics  for  Engineering  and  the 
Sciences .  3rd  ed.  San  Francisco,  CA:  Dellen  Publishing  Company,  1992.  pp.  30-31. 

36  Entire  paragraph  drawn  from  Mendenhall,  William  and  Terry  Sincich.  Statistics  for  Engineering  and  the 
Sciences .  3rd  ed.  San  Francisco,  CA:  Dellen  Publishing  Company,  1992.  pp.  30-31. 

37  Entire  paragraph  drawn  from  Mendenhall,  William  and  Terry  Sincich.  Statistics  for  Engineering  and  the 
Sciences.  3rd  ed.  San  Francisco,  CA:  Dellen  Publishing  Company,  1992.  p.  31. 
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Skewness 

Skewness  characterizes  the  degree  of  asymmetry  of  a  distribution  around  its  mean.  Positive 
skewness  indicates  a  distribution  with  an  asymmetric  tail  extending  toward  more  positive 
values.  Negative  skewness  indicates  a  distribution  with  an  asymmetric  tail  extending  toward 
more  negative  values.38  The  equation  for  skewness  is: 


skewness  = 


n 

(«-!)(«-  2) 


(4) 


Kurtosis 

Kurtosis  characterizes  the  relative  peakedness  or  flatness  of  a  distribution  compared  with  the 
normal  distribution.  Positive  kurtosis  indicates  a  relatively  peaked  distribution.  Negative 
kurtosis  indicates  a  relatively  flat  distribution.39  The  equation  for  kurtosis  is: 


kurtosis  - 


n(n  + 1) 

(/j-1)(m-2)(w-3) 


3(»~1)2 

(m-2)(m-3) 


(5) 


Measures  of  Relative  Standing 

Measures  of  relative  standing  are  the  third  group  of  numerical  descriptive  measures.  They 
are  a  collection  of  mathematical  techniques  used  to  describe  the  relative  position  of  an 
observation  within  the  data  set.  In  certain  situations,  one  can  gain  additional  information 
about  an  observation  based  on  where  it  falls  in  the  overall  distribution.  This  section  reviews 
the  various  measures  of  relative  standing  that  are  used  throughout  the  discrepancy  analysis. 


Percentile 

The  1 00pth  percentile  of  a  data  set  is  a  value  ofy  located  so  that  1 00p%  of  the  area  under  the 
relative  frequency  distribution  for  the  data  lies  to  the  left  of  the  100/>,h  percentile  and  the 
remaining  100(1  -  p)%  of  the  area  lies  to  its  right.40  The  median  of  a  given  data  set  is  its  50th 


38  Entire  paragraph  drawn  from  Microsoft  Excel  2000  Function  Reference.  Redmond,  WA:  Microsoft 
Corporation,  2000. 

39  Entire  paragraph  drawn  from  Microsoft  Excel  2000  Function  Reference.  Redmond,  WA:  Microsoft 
Corporation,  2000. 

40  Mendenhall,  William  and  Terry  Sincich.  Statistics  for  Engineering  and  the  Sciences.  3rd  ed.  San  Francisco, 
CA:  Dellen  Publishing  Company,  1992.  pp.  34-35. 
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th  th 

percentile,  and  the  lower  and  upper  quartiles  are  defined  as  the  25  and  75  percentiles, 
respectively.  The  quartile  points,  and  any  percentiles  in  general,  are  generally  not  as  crisply 
defined  for  small  data  sets. 

Sample  Z-Score 

The  z-score  for  a  value  y  of  a  data  set  is  the  distance  that  y  lies  above  or  below  the  mean, 
measured  in  units  of  the  standard  deviation: 


s 

By  definition,  the  z-score  describes  the  location  of  an  observation  y  relative  to  the  mean. 
Negative  z-scores  indicate  that  the  observation  lies  to  the  left  of  the  mean;  positive  z-scores 
indicate  that  the  observation  lies  to  the  right  of  the  mean.  According  to  the  Empirical  Rule, 
most  of  the  observations  in  a  data  set  will  be  less  than  2  standard  deviations  from  the  mean 
and  will  therefore  have  z-scores  less  than  2  in  absolute  value.  In  addition,  almost  all 
observations  will  be  within  3  standard  deviations  of  the  mean  and  will  have  z-scores  less  than 
3  in  absolute  value.41 

Outliers 

An  observation  y  that  is  unusually  large  or  small  relative  to  the  other  values  in  a  data  set  is 
called  an  outlier.  Outliers  typically  are  attributable  to  one  of  the  following  causes: 

•  The  measurement  is  observed,  recorded,  or  entered  into  the  computer  incorrectly. 

•  The  measurement  comes  from  a  different  population. 

•  The  measurement  is  correct,  but  represents  a  rare  chance  event. 

The  most  obvious  method  for  determining  whether  an  observation  is  an  outlier  is  to  calculate 
its  z-score.  Observations  with  z-scores  greater  than  3  in  absolute  value  are  considered 
outliers.  However,  the  presence  of  one  or  more  outliers  in  a  data  set  can  inflate  the  value  of  s 
used  to  calculate  the  z-score.  Consequently,  it  will  be  less  likely  that  an  errant  observation 
would  have  a  z-score  larger  than  3  in  absolute  value.  Another  method  for  determining 
whether  an  observation  is  an  outlier  is  calculating  the  interquartile  range,  IQR,  which  is  the 
difference  between  the  upper  and  lower  quartile  values.  Observations  less  than  1 .5(IQR) 

41  Entire  paragraph  drawn  from  Mendenhall,  William  and  Terry  Sincich.  Statistics  for  Engineering  and  the 
Sciences.  3rd  ed.  San  Francisco,  CA:  Dellen  Publishing  Company,  1992.  p.  36. 
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below  the  lower  quartile,  or  greater  than  1.5(IQR)  above  the  upper  quartile  are  suspect 
outliers.  Those  less  than  3(IQR)  below  the  lower  quartile,  or  greater  than  3(IQR)  above  the 
upper  quartile  are  highly  suspect  outliers.  In  contrast  with  z-scores,  the  values  of  the 
quartiles  used  to  calculate  IQR  are  not  affected  by  the  presence  of  outliers.42 

Box  Plot 

The  box  plot  display  is  useful  for  graphically  examining  the  dispersion  of  the  data  set.  It  was 
designed  by  John  Tukey,  and  is  a  graphical  display  that  indicates  range,  quartiles,  inter¬ 
quartile  range,  median,  and  outliers  of  a  data  set.  An  annotated  sketch  of  a  box  plot  is  shown 
in  Figure  6.  The  bold  horizontal  line  in  the  middle  of  the  box  indicates  the  sample  median. 

A  single  filled  square  designates  the  sample  mean.  The  edges  of  each  box,  called  hinges, 
mark  the  25th  and  75th  percentiles,  so  that  the  central  50%  of  the  data  values  fall  within  the 
range  of  the  box.  The  whiskers,  or  the  vertical  lines  extending  up  and  down  from  each  box, 
show  the  range  of  values  that  fall  within  1.5(IQR)  of  the  hinges.  Data  points  that  have  values 
between  1 .5(IQR)  and  3(IQR)  outside  the  hinges  are  marked  by  an  open  circle,  to  designate  a 
suspect  outlier.  Data  points  more  than  3  (IQR)  below  the  lower  hinge  or  above  the  upper 
hinge  are  marked  by  an  asterisk,  to  designate  a  highly  suspect  outlier.43 


42  Entire  paragraph  drawn  from  Mendenhall,  William  and  Terry  Sincich.  Statistics  for  Engineering  and  the 
Sciences.  3rd  ed.  San  Francisco,  CA:  Dellen  Publishing  Company,  1992.  pp.  37-39. 

43  Entire  paragraph  drawn  from  SPSS  Base  10.0  Applications  Guide.  Chicago,  IL:  SPSS,  Inc.,  1999.  pp.  40-41. 
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Figure  6  -  Annotated  Diagram  of  a  Box  Plot 
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Measures  of  Correlation 

Measures  of  correlation  are  the  fourth  and  final  group  of  numerical  descriptive  measures 
used  in  this  study.  They  are  a  collection  of  mathematical  techniques  used  to  describe  the 
relationship  between  two  variables  in  a  set  of  observations.  As  before,  each  technique  has 
strengths  and  weaknesses  that  make  it  more  useful  and  accurate  in  some  cases,  but  not  others. 
The  ‘best’  measure  of  correlation  for  a  given  data  set  depends  both  on  the  type  of  descriptive 
information  desired  and  the  nature  of  the  data  being  evaluated. 

Spearman ’s  Rank  Correlation  Coefficient 

Since  the  discrepancy  data  does  not  exhibit  a  normal  distribution,  a  correlation  method 
designed  for  this  type  of  data  must  be  used.  In  nonparametric  regression,  tests  of  model 
adequacy  do  not  require  any  assumptions  about  the  distribution  of  the  random  error  term; 
thus,  they  are  distribution-free.  Spearman’s  rank  correlation  coefficient  can  be  used  to  test 
for  correlation  between  two  variables,  y  and  x.  It  is  found  by  first  ranking  the  values  of  each 
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variable  separately.  Then  the  coefficient  is  calculated  using  the  sum  of  squares  of  the 
deviations  for  the  rankings.44 

The  correlation  coefficient  provides  a  scaleless  quantitative  measure  of  the  strength  of  the 
linear  relationship  between  x  and  y.  The  value  of  the  coefficient  is  always  between  -1  and 
+1,  regardless  of  the  units  of  the  variables.  Values  near  0  imply  little  or  no  relationship 
between  the  variables,  while  values  near  -1  or  +1  indicate  a  strong  relationship.  The  values 
of — 1  and  +1  themselves  correspond  to  a  situation  where  all  the  data  points  fall  exactly  on  the 
least  squares  line.  Positive  values  imply  that  y  increases  as  x  increases;  negative  values 
imply  that  y  decreases  as  x  increases.45 

It  is  important  to  note  that  a  strong  correlation  between  two  variables  does  not  necessarily 
imply  causality  between  one  and  the  other.  It  merely  indicates  a  mathematical  relationship 
between  the  observed  values  of  one  variable  and  the  observed  values  of  the  other. 

Nonparametric  Test  for  Rank  Correlation 

It  is  not  enough  to  state  that  a  correlation  exists  between  two  variables  of  interest.  One  must 
also  determine  the  likelihood  of  a  similar  correlation  occurring  by  chance.  Generally,  as  the 
number  of  observations  in  the  data  set  increases,  the  less  likely  a  similar  correlation  could  be 
found  by  chance.  Therefore,  each  time  the  correlation  coefficient  is  calculated,  it  must  be 
compared  to  a  pre-defined  critical  value  for  two-tailed  significance.  In  this  study,  only 
correlations  larger  than  the  critical  value  for  a  two-tailed  significance  of  0.01  were  accepted. 

Analysis  Method 

The  data  collection  procedure  outlined  earlier  in  the  chapter  resulted  in  a  database  of  over 
9,200  discrepancies,  all  categorized  according  to  related  subsystem  and  root  cause.  In 
addition,  each  spacecraft  was  categorized  by  several  unique  design  parameters,  which  then 
became  associated  with  the  corresponding  discrepancies  for  that  spacecraft.  This  section 

44  Portions  drawn  from  Mendenhall,  William  and  Terry  Sincich.  Statistics  for  Engineering  and  the  Sciences. 

3rd  ed.  San  Francisco,  CA:  Dellen  Publishing  Company,  1992.  p.784. 

45  Portions  drawn  from  Mendenhall,  William  and  Terry  Sincich.  Statistics  for  Engineering  and  the  Sciences. 

3rd  ed.  San  Francisco,  CA:  Dellen  Publishing  Company,  1992.  p.  441. 
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describes  the  steps  taken  to  analyze  the  compiled  data  set  and  obtain  the  results  presented  in 
Chapter  6. 

Category  Relative  Frequency 

For  data  that  is  divided  into  discrete  groups  or  categories,  the  category  relative  frequency  is 
the  proportion  of  the  total  number  of  observations  that  fall  into  each  category. 46  This  is  the 
case  with  discrepancy  data  when  categorized  according  to  affected  subsystem  and  root  cause. 
The  category  relative  frequency  diagram  is  similar  to  a  histogram,  which  is  used  to  depict 
absolute  or  relative  frequencies  of  numerically  based  data. 

Two  category  relative  frequency  diagrams  were  created  for  the  subsystem  analysis,  one 
diagram  showing  the  number  of  discrepancies  occurring  in  each  individual  subsystem,  and 
the  second  diagram  showing  the  number  of  discrepancies  occurring  in  spacecraft  subsystems 
vice  infrastructure-related  subsystems.  The  few  database  entries  that  were  associated  with 
multiple  subsystems  were  excluded  from  this  depiction,  since  each  observation  must  fall  into 
one  and  only  one  category. 

Spacecraft  Normalization 

As  mentioned  previously,  operations  facilities  have  their  own  procedures  for  recording 
discrepancy  data.  This  leads  to  large  variations  in  the  number  of  discrepancies  provided  by 
each  organization  as  well  as  the  detail  provided  in  each  report  entry.  This  can  be  problematic 
when  merging  the  reports  into  one  data  set  for  analysis,  since  organizations  that  contribute 
smaller  data  sets  tend  to  get  lost  in  the  noise  of  organizations  that  contribute  very  large 
quantities  of  reports.  In  addition,  the  presence  of  one  very  large  and  unique  data  set  can 
potentially  skew  the  results  of  the  entire  study. 

One  solution  to  this  problem,  and  the  one  chosen  for  use  here,  is  to  normalize  the  discrepancy 
data  on  the  basis  of  each  spacecraft.  This  is  accomplished  by  dividing  the  number  of 
discrepancies  associated  with  each  subsystem  or  cause  for  a  given  spacecraft  by  the  sum  total 
of  the  spacecraft’s  discrepancies.  The  resulting  data  observations  for  the  spacecraft  are  a  set 

46  Mendenhall,  William  and  Terry  Sincich.  Statistics  for  Engineering  and  the  Sciences.  3rd  ed.  San  Francisco, 
CA:  Dellen  Publishing  Company,  1992.  p.  14. 
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of  percentages,  one  for  each  subsystem  or  cause,  which  describes  the  distribution  of 
problems  among  the  various  categories. 


Figure  7  is  an  example  database  display  for  a  notional  spacecraft.  It  shows  how  the  various 
design  elements  are  recorded  for  each  spacecraft,  and  how  the  number  of  discrepancy  reports 
in  each  associated  subsystem  category  and  root  cause  category  is  reported.  As  an  example, 
the  normalized  percentage  score  of  the  Communications  Relay  subsystem  for  the  notional 
spacecraft  in  Figure  7  would  be  1 10/  300  =  37%. 


Figure  7  -  Database  Display  for  Notional  Spacecraft 
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Percentile  Rankings 

The  result  of  the  normalization  process  is  a  list  of  percentages  representing  the  fraction  of 
discrepancies  for  each  spacecraft  that  are  associated  with  each  subsystem  category.  Table  1 
is  a  notional  depiction  of  a  portion  of  the  normalized  discrepancy  data. 


Table  1  -  Normalized  Discrepancy  Data  for  Notional  Spacecraft 
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Note  that  for  the  actual  normalized  data  set,  the  list  includes  all  spacecraft  in  the  database  and 
all  subsystem  categories  used  in  the  study.  Each  complete  row  sums  to  100%  -  representing 
the  total  discrepancies  for  a  given  spacecraft.  Meanwhile,  each  column  can  be  analyzed  to 
determine  whether  or  not  the  particular  subsystem  is  consistently  identified  as  a  problem 
across  multiple  spacecraft  programs.  Thus,  the  column  of  percentage  scores  for  each 
subsystem  becomes  the  basis  of  the  statistical  analysis  described  next.  The  root  cause 
category  percentage  rankings  are  treated  in  a  similar  fashion. 

Statistical  Measures 

When  the  column  of  subsystem  percentage  scores  becomes  the  basis  of  analysis,  a  histogram 
is  used  to  show  the  number  of  observations  that  fall  within  each  range  of  percentage  values. 
Figure  8  is  an  example  of  such  a  histogram  for  a  set  of  notional  observations  in  the 
Communications  Relay  subsystem.  The  example  value  calculated  above,  37%,  appears  in 
the  figure  as  the  only  observed  value  falling  in  the  range  of  35  -  40%. 
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Figure  8  -  Notional  Histogram  of  Percentage  Scores  for  the  Comm  Relay  Subsystem 
5-1 - 1 


Range  of  Percentages 


The  set  of  observations  depicted  in  Figure  8  can  be  analyzed  using  the  statistical  techniques 
described  above.  The  resulting  parameters,  shown  in  Table  2,  specify  that  the  sample  mean 
is  14.41%.  This  indicates  that  on  average,  14.41%  of  the  discrepancies  reported  for  each 
spacecraft  in  the  data  set  were  associated  with  the  communications  subsystem. 


Table  2  -  Statistical  Parameters  for  Notional  Comm  Relay  Subsystem 


Mean 

Median 

Trim  Mean 

Max 

Min 

Range 

Variance 

Std  Dev 

Skewness 

Kurtosis 

Gomm  Relay 

0.1244 

0.1000 

0.1244 

0.3700 

0.0500 

0.3200 

0.0125 

0.1118 

1.7349 

2.2968 

The  other  parameters  in  Table  2  offer  additional  insight.  The  sample  median,  10%,  reflects 
that  a  large  number  of  observations  in  the  data  set  were  less  than  the  sample  mean.  This  is 
supported  by  the  skewness  value,  which  represents  a  distribution  with  a  longer  right 
(positive)  tail. 

One  disadvantage  to  the  normalization  process  is  that  it  reduces  an  extremely  large  data  set 
of  discrepancy  reports  down  to  a  data  set  with  the  same  number  of  observations  as  the 


58 


David  L.  Ferris 


Chapter  5  -  Methods  and  Procedures 


number  of  spacecraft.  Although  a  reduced  data  set  is  easier  to  deal  with  computationally,  the 
smaller  sample  size  can  make  statistical  parameters  less  representative. 

Correlation  Analysis 

The  final  step  of  the  analysis  is  to  check  for  correlations  between  the  spacecraft  design 
elements  listed  in  Chapter  4  and  the  type  of  discrepancies  that  occur  during  operations.  The 
normalized  discrepancy  data,  similar  to  that  shown  in  Table  1,  is  first  grouped  according  to 
like  values  for  one  particular  design  element  of  interest,  for  example  mission  type.  The 
groupings  are  then  arranged  in  ascending  order  and  ranked.  Likewise,  the  percentages  of 
discrepancies  for  a  subsystem  of  interest,  like  communications  relay,  are  also  ranked. 
Spearman’s  rank  correlation  coefficient  is  then  calculated  for  the  two  sets  of  rankings  and 
compared  to  the  critical  value  for  two-tailed  significance. 

This  process  was  repeated  for  each  combination  of  design  element  and  subsystem  category 
and  each  combination  of  design  element  and  root  cause  category.  Combinations  with  a 
Spearman’s  rank  correlation  coefficient  greater  than  the  critical  value  are  reported  in  Chapter 
6,  while  those  weaker  than  the  critical  value  were  discarded. 
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Chapter  6  -  Results 

This  chapter  presents  the  results  of  the  analysis  described  in  Chapter  5  when  applied  to  the 
discrepancy  database.  The  general  characteristics  of  the  data  are  reviewed  first,  followed  by 
the  results  pertaining  to  each  hypothesis.  Additional  observations  based  on  the  root  cause 
information  are  presented  at  the  end  of  the  chapter. 

Review  of  Data 

The  database  compiled  for  this  study  consisted  of  over  9,200  discrepancy  reports  collected 
during  operation  of  1 1  on-orbit  spacecraft.  The  mission  areas  represented  by  the  satellites  in 
the  study  included  communications,  remote  sensing,  research  and  development,  and 
observatories.  The  breakdown  of  mission  areas  is  depicted  graphically  in  Figure  9. 


Figure  9  -  Breakdown  of  Spacecraft  by  Mission  Area 
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The  dates  on  which  the  discrepancies  occurred  ranged  from  June  1992  to  March  2001.  The 
distribution  of  discrepancies  as  well  as  the  number  of  operational  spacecraft  in  the  database 
throughout  the  time  period  is  shown  in  Figure  10. 


Figure  10  -  Distribution  of  Discrepancies  Reported  by  Date 


Several  civil  and  military  organizations  contributed  data  for  this  analysis.  The  breakdown  of 
all  spacecraft  according  to  the  organizations  responsible  for  operating  them  is  depicted 
graphically  in  Figure  11.  It  is  important  to  note  that  there  are  no  commercial  systems 
represented  in  this  study. 
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Figure  11  -  Breakdown  of  Spacecraft  by  Responsible  Organization 
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The  nature  and  validity  of  the  sampling  of  spacecraft  included  in  this  study  is  discussed 
further  in  Chapter  7. 

First  Hypothesis 

The  first  hypothesis  proposed  in  this  study  was  that  most  problems  encountered  by  an 
operator  do  not  involve  the  spacecraft  at  all,  but  are  attributed  to  other  elements  of  the  space 
system.  The  most  direct  way  to  address  the  issue  is  to  report  the  number  of  discrepancies 
belonging  to  each  subsystem  category.  This  is  accomplished  with  the  category  relative 
frequency  diagram  shown  in  Figure  12.  The  diagram  indicates  that  the  two  subsystems  most 
frequently  associated  with  discrepancies  are  ground  systems  and  communications  relays. 
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Figure  12  -  Percentage  of  All  Discrepancies  Reported  vs.  Associated  Subsystem 


Associated  Subsystems 


The  next  step  is  to  group  the  various  subsystems  according  to  whether  they  are  located  on  the 
spacecraft  or  within  the  operational  infrastructure.  The  subsystems  located  on,  or  associated 
with,  the  spacecraft  include:  EPDS,  GNC,  payload,  propulsion,  SMS,  DMS,  TT&C,  thermal, 
harness,  other  spacecraft,  and  spacecraft.  The  subsystems  considered  part  of  the  operational 
infrastructure  include:  ground  systems,  communications  relay,  tracking  equipment,  facilities, 
administrative  equipment,  and  other  infrastructure.  Detailed  explanations  of  the  individual 
categories  can  be  found  in  Chapter  4.  The  percentage  of  all  discrepancies  reported  against 
each  major  grouping  -  spacecraft  or  infrastructure  -  is  shown  in  the  category  relative 
frequency  diagram  in  Figure  13. 
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Figure  13  -  Percentage  of  All  Discrepancies  Reported  by  Major  Grouping 
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Figure  13  shows  that  87%  of  all  discrepancies  are  associated  with  subsystems  found  in  the 
operational  infrastructure,  a  clear  majority  that  tends  to  support  the  hypothesis.  However,  the 
category  relative  frequency  diagram  can  be  skewed  if  one  or  two  outlier  spacecraft  have  an 
uncommonly  large  number  of  discrepancies  associated  with  infrastructure.  The 
normalization  process  outlined  in  Chapter  5  is  designed  to  overcome  this  limitation  and 
provide  a  more  accurate  representation  of  the  data  set. 

The  result  of  the  normalization  process  is  a  list  of  percentages  representing  the  fraction  of 
discrepancies  for  each  spacecraft  that  are  associated  with  each  subsystem  category.  The 
percentages  in  each  column  can  be  analyzed  to  determine  whether  or  not  the  particular 
subsystem  is  consistently  identified  as  a  problem  across  multiple  spacecraft  programs.  The 
statistical  parameters  described  earlier  in  Chapter  5  are  used  for  this  purpose.  Table  3  shows 
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the  resulting  statistics  for  each  subsystem  against  which  at  least  one  discrepancy  was 
reported. 


Table  3  -  Resulting  Statistics  for  Each  Subsystem  47 


Mean  s 

Median 

Trim  Mean 

Max 

Min 

Range 

Variance 

Std  Dev 

Skewness 

Kurtosis 

Groufilystem 

0.6578 

0.8378 

0.6578 

0.9450 

0.2058 

0.7392 

0.0997 

0.3157 

-0.6623 

-1.7746 

Comm  Relay 

n 

0.0018 

0.5279 

Kims 

0.0005 

-0.5104 

-1.8422 

0.0000 

0.0740 

2.9946 

0.0000 

0.0000 

0.0129 

0.0000 

0.0052 

0.8267 

-1.2694 

^§1  1121  ^  | 

0.0000 

0.1579 

0.0000 

0.1579 

0.0476 

3.3166 

11.0000 

iPiS*  ks 

0.0200 

0.0000 

0.0200 

0.0865 

0.0000 

0.0865 

0.0010 

0.0314 

1.3630 

0.4639 

0.0000 

0.0000 

0.2105 

0.0059 

0.0768 

1.5572 

1.1334 

0.0000 

0.0218 

0.0000 

0.1540 

0.0022 

0.0470 

2.6472 

7.3777 

£ 

0.0221 

0.0000 

0.0221 

0.1053 

:  '  .  j 

0.0034 

0.0000 

0.0034 

0.0193 

Emus 

EB 

Slim- 

0.0014 

0.0000 

0.0014 

0.0072 

KE39 

SMS  ...  .1 

0.0008 

0.0000 

0.0008 

KSKifSSl 

0.0000 

0.0018 

4.2997 

mil 

0.0003 

0.0000 

EEEH3I 

0.0000 

0.0010 

10.9074 

Overall  Spacecraft 

0.0003 

0.0000 

0.0003 

0.0032 

0.0000 

0.0032 

0,0000 

0.0010 

3.3166 

11.0000 

Table  3  contains  a  great  deal  of  information  about  each  subsystem  category.  Using  the  first 
row  as  an  example,  an  average  65%  of  discrepancies  reported  for  a  given  spacecraft  are 
associated  with  the  ground  system.  The  median  value  indicates  that  there  is  a  fairly  large 
concentration  of  spacecraft  in  the  sample  for  which  the  ground  system  accounts  for  a  large 
number  (~84%)  of  discrepancies.  The  maximum,  minimum,  and  skewness  values  indicate 
that  at  least  one  spacecraft  with  a  low  number  of  ground  system  problems  (~21%)  is  pulling 
down  the  mean  and  skewing  the  distribution  of  percentages  downward. 


Unfortunately,  the  detailed  information  shown  in  Table  3  is  difficult  to  visualize  and 
interpret.  Similar  information  can  be  presented  graphically  in  the  form  of  a  box  plot.  Figure 
14  shows  the  distribution  of  percentages  for  each  subsystem  category  using  a  box  plot. 


It  is  less  obvious  in  Figure  14  than  in  Figure  13  that  the  operational  infrastructure  accounts 
for  a  much  larger  percentage  of  discrepancies  than  spacecraft  subsystems.  However,  it  gives 

47  Only  subsystem  categories  with  at  least  one  discrepancy  reported  are  shown  in  this  table.  The  values  listed  in 
the  first  five  columns  are  decimal  percentages. 
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a  more  accurate  presentation  of  the  fact  that  two  elements  of  the  operational  infrastructure  in 
particular  -  ground  systems  and  communications  relays  -  are  consistently  attributed  to 
discrepancies  across  several  different  spacecraft  programs. 


Figure  14  -  Box  Plot  of  Subsystem  Statistics 


Second  Hypothesis 

The  second  hypothesis  proposed  in  this  study  was  that  correlations  exist  between  aspects  of  a 
space  system  design  and  the  nature  of  problems  experienced  by  the  operations  staff  over  the 
long  term.  To  address  this  question,  correlation  tests  were  conducted  for  each  combination 
of  design  element  and  subsystem  category.  Combinations  with  a  Spearman’s  rank 
correlation  coefficient  greater  than  the  critical  value  are  reported  below,  while  those  weaker 
than  the  critical  value  were  discarded. 
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The  correlations  between  most  of  the  subsystem  categories  were  not  statistically  significant 
based  on  the  sample  size  and  the  number  of  discrepancies  reported  for  each  category.  Two 
subsystem  categories,  in  particular,  did  display  a  strong  correlation  with  elements  of  the 
spacecraft  design:  ground  system  and  communications  relay.  The  percentage  of  each 
satellite’s  discrepancies  associated  with  each  of  these  two  subsystems  is  listed  in  Table  4. 


Table  4  -  Percent  of  Each  Satellite's  Discrepancies  Associated  with  Ground  System  and  Comm  Relay 
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0% 

The  first  correlation  presented  here  is  between  the  mission  type  of  the  spacecraft  and  the 
percentage  of  its  discrepancies  associated  with  the  ground  system.  The  Spearman’s  rank 
correlation  coefficient  for  this  relationship  is  0.852.  Figure  15  shows  the  percentage  of 
discrepancies  associated  with  the  ground  system  reported  by  each  spacecraft  plotted  against 
the  mission  type  of  the  spacecraft.  The  correlation  indicates  that  the  percentage  of  problems 
associated  with  the  ground  system  generally  changes  given  a  certain  mission  type.  In  other 
words,  spacecraft  performing  certain  missions  tend  to  experience  ground  system  problems 
more  frequently  than  spacecraft  performing  other  missions.  However,  it  is  important  to  note 
that  the  presence  of  a  correlation  does  not  imply  that  certain  spacecraft  mission  types  cause  a 
larger  percentage  of  ground  system  problems  than  others. 
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Figure  15  -  Ground  System  Discrepancies  per  Spacecraft  vs.  Mission  Type 


The  second  correlation  presented  here  is  between  the  organization  and  the  percentage  of 
discrepancies  associated  with  the  communications  relay.  The  Spearman’s  rank  correlation 
coefficient  for  this  relationship  is  0.850.  Figure  16  shows  the  percentage  of  discrepancies 
associated  with  the  communications  relay  reported  by  each  spacecraft  plotted  against  the 
organization  responsible  for  operating  the  spacecraft.  In  this  case,  organization  refers  to  a 
group  or  team  that  operates  one  or  more  spacecraft.  The  correlation  indicates  that  the 
percentage  of  problems  associated  with  the  communications  relay  generally  changes  from 
one  organization  to  another.  Once  again,  it  is  important  to  note  that  the  presence  of  a 
correlation  does  not  imply  causality. 
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Figure  16  -  Communications  Relay  Discrepancies  per  Spacecraft  vs.  Organization 


Thus,  the  data  support  the  second  hypothesis,  in  that  a  mathematical  relationship  exists 
between  certain  spacecraft  design  elements  and  the  nature  of  operational  problems 
experienced  in  the  long  run.  However,  the  analysis  cannot  be  considered  exhaustive,  due  to 
the  presence  of  multiple  variables  and  a  sample  set  which  does  not  fully  explore  the  values 
for  each  variable  in  an  independent  fashion.  This  issue  is  discussed  further  in  Chapter  7. 

Additional  Observations 

Most  of  the  discrepancy  data  sets  contributed  for  this  study  contained  root  cause  information 
in  each  report.  Those  that  did  not  were  excluded  from  this  portion  of  the  analysis.  In  a 
similar  fashion  as  the  subsystem  category,  the  first  step  is  to  report  the  number  of 
discrepancies  belonging  to  each  subsystem  category.  This  is  accomplished  with  the  category 
relative  frequency  diagram  shown  in  Figure  17.  The  diagram  indicates  that  software 
problems  are  the  most  frequent  cause  of  discrepancies,  occurring  in  61%  of  the  reported 
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cases.  The  equipment,  unknown,  procedure,  and  no  anomaly  categories  were  comparable  at 
approximately  7%  each.  As  a  reminder,  “software  problems”  are  defined  as  discrepancies 
caused  by  software,  either  on  the  spacecraft  or  on  the  ground  equipment.  Includes  problems 
caused  by  hung  processes,  or  instances  where  a  computer  reboot  is  required  to  restore 
functionality.  Categories  not  shown  on  the  graph  were  not  reported  on  any  of  the 
discrepancies  in  the  data  set. 


Figure  17  -  Percentage  of  All  Discrepancies  Reported  vs.  Root  Cause 


Root  Cause 


The  normalization  process  can  also  be  applied  to  the  root  cause  analysis  to  prevent  one  or 
two  spacecraft  from  skewing  the  results.  When  the  data  is  normalized  on  a  per-spacecraft 
basis,  the  resulting  statistical  parameters  for  each  root  cause  category  are  summarized  below 
in  Table  5.  To  make  the  data  in  Table  5  easier  to  interpret,  it  is  shown  graphically  using  a 
box  plot  in  Figure  18.  The  diagram  shows  that  software  is  the  only  root  cause  category 
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consistently  reported  as  a  frequently  occurring  problem  among  all  of  the  spacecraft  in  the 
data  set. 


Table  5  -  Resulting  Statistics  for  Each  Root  Cause  48 


Mean 
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Figure  18  -  Box  Plot  of  Root  Cause  Statistics 


Root  Cause 


48  Only  root  cause  categories  with  at  least  one  discrepancy  reported  are  shown  in  this  table.  The  values  listed  in 
the  first  five  columns  are  decimal  percentages. 


David  L.  Ferris 


71 


Characterization  of  Operator-Reported  Discrepancies  in  Unmanned  On-Orbit  Space  Systems 

Figure  19  provides  more  insight  into  the  nature  of  discrepancies  documented  as  caused  by 
software.  Over  half  of  the  discrepancies  caused  by  software  occur  in  one  of  the  ground 
system  components.  All  told,  91%  are  associated  with  components  in  the  infrastructure  and 
9%  are  associated  with  components  on  board  the  spacecraft. 


The  root  cause  analysis  provides  insight  as  to  the  various  sources  of  discrepancies  reported 
from  several  organizations.  Additional  discussion  concerning  these  results  is  included  in 
Chapter  7. 
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Chapter  7  -  Discussion  and  Conclusions 

Several  aspects  of  the  results  presented  in  Chapter  6  are  analyzed  in  this  chapter.  The  first 
section  is  an  overview  of  the  costs  of  discrepancies.  The  second  section  is  a  discussion  of  the 
data  sampling  technique  and  the  methodology  used  in  this  study,  as  well  as  general 
observations  about  the  data  collected.  The  final  section  summarizes  the  analysis  and  presents 
several  recommendations  for  the  use  and  future  extension  of  this  research. 


Cost  of  Discrepancies 

Although  the  occurrences  of  discrepancies  are  well  documented  within  most  organizations, 
the  associated  costs  of  dealing  with  those  discrepancies  are  rarely  collected.  There  are 
several  reasons  why  this  is  the  case: 

•  Cost  information  does  not  come  from  one  source.  Discrepancy  cost  information 
typically  includes  the  number  of  labor  hours  that  several  different  individuals  spend 
in  engineering  meetings  and  working  groups;  the  number  of  labor  hours  a  handful  of 
individuals  spend  researching,  troubleshooting,  and  repairing;  and  the  direct  expenses 
for  equipment  and  materials.  In  order  to  get  a  complete  and  accurate  representation 
of  discrepancy  costs,  everyone  and  everything  involved  in  the  process  must  be 
accounted  for. 

•  Ambiguous  nature  of  discrepancy  costs.  The  delineation  between  work  necessary 
to  accomplish  a  routine  task  and  work  spent  troubleshooting  a  specific  problem  is 
difficult  to  establish  in  a  formal  fashion.  In  some  cases,  operators  may  expend 
significant  effort  troubleshooting  a  problem  before  the  associated  discrepancy  report 
is  even  written.  In  addition,  work  spent  troubleshooting  one  particular  problem 
sometimes  serves  to  correct  one  or  more  other  problems.  Trying  to  assign  cost  to 
multiple  discrepancies  can  be  difficult  and  subjective. 

•  Lack  of  reporting  mechanism.  Even  if  individual  employees  track  the  effort  they 
expend  on  discrepancies,  there  must  be  a  mechanism  in  place  to  record  and  collect  the 
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information.  Accounting  systems  in  commercial  organizations  are  usually  designed 
to  collect  cost  data  on  a  contract  basis,  rather  than  a  functional  basis.  Government 
organizations  -  whose  employees  are  salaried  -  rarely  collect  labor  data  at  all,  and 
have  no  infrastructure  in  place  to  do  so. 

•  Cost  of  collecting  cost  data  itself.  Tracking,  recording,  reporting,  and  compiling 
cost  information  takes  time  and  effort.  As  the  number  of  functional  tasks  (including 
discrepancies)  and  the  number  of  employees  increase,  the  work  necessary  to  collect 
cost  data  also  increases.  For  larger  organizations,  it  can  easily  require  one  or  more 
full-time  employees  just  to  deal  with  the  information.  Most  organizations  choose, 
justifiably,  not  to  pay  this  additional  cost. 

•  No  desire  or  need  for  discrepancy  cost  data.  In  some  organizations, 
troubleshooting  problems  is  considered  part  of  the  job,  so  there  is  no  need  to 
determine  exactly  how  much  effort  is  spent  resolving  problems  versus  effort  spent 
performing  routine  duties.  In  addition,  discrepancy  cost  data  is  not  useful  when  an 
organization  does  not  even  track  the  discrepancies  themselves. 

Despite  the  lack  of  explicit  discrepancy  cost  information,  it  is  possible  to  estimate  the  cost  of 
dealing  with  problems.  Interviews  with  the  operational  supervisors  in  two  of  the 
participating  organizations  yielded  fairly  consistent  estimates  of  the  labor  required  to 
troubleshoot  a  problem.  Those  estimates  are  used  in  the  following  cost  analysis. 

Generally,  discrepancies  can  be  divided  into  two  categories:  repetitive  or  relatively 
insignificant  problems,  and  significant  problems.  The  percentage  of  significant  problems  is 
roughly  equal  to  the  percentage  of  spacecraft-related  problems  encountered  on-orbit.49  Using 
this  rule  of  thumb  for  all  of  the  spacecraft  included  in  this  study,  approximately  1200 
problems  were  considered  significant  and  the  remaining  8000  were  routine. 

49  Most  spacecraft-related  problems  are  -  or  are  treated  as  -  significant.  Repetitive  spacecraft-related  problems, 
once  well  characterized,  tend  to  receive  more  cursory  treatment  or  are  dealt  with  using  contingency  procedures. 
Therefore,  they  can  be  considered  routine.  However,  the  non-spacecraft  infrastructure  also  occasionally 
experiences  significant  problems  that  must  be  accounted  for.  Thus,  the  percentage  of  problems  that  are 
spacecraft-related  is  a  good  approximation  for  the  percentage  of  problems  that  are  significant. 
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Routine  problems  are  typically  dealt  with  first  in  a  daily  or  weekly  status  meeting  with  6-10 
people  in  attendance.  The  group  review  takes  approximately  10  minutes  per  discrepancy, 
and  the  problem  will  be  assigned  to  one  or  two  individuals  to  investigate  and/or  resolve.  The 
average  time  spent  researching,  troubleshooting,  and  resolving  a  particular  problem  and 
implementing  the  solution  is  8  hours,  at  which  point  the  results  will  be  reviewed  at  the  status 
meeting  for  another  10  minutes  before  being  closed.  This  yields  a  fully  burdened  per- 
discrepancy  cost,  not  including  equipment  or  materials,  of: 

/  \  f  Speople']  f 0.1 61hours\  /OJ  \  (  $75 

(2 meetings)*.  — — - —  x  - I  +  [Shours )  x  - - 

meeting J  ^  person  )  \hour 


Significant  problems,  on  the  other  hand,  usually  involve  immediate  diagnosis,  safing, 
notification,  and  documentation  by  at  least  one  operator  and  one  on-call  engineer  for 
approximately  3  hours  each.  The  status  meeting  attendees  receive  an  initial  briefing  on  the 
situation  for  approximately  20  minutes.  A  team  of  4-6  individuals  spends  roughly  8  hours 
each,  not  necessarily  consecutive,  researching  and  troubleshooting  the  problem  and 
developing  a  solution.  Two  people  each  spend  an  additional  3  hours  implementing  and 
testing  the  solution  and  documenting  the  results.  The  results  are  reviewed  at  the  status 
meeting  for  approximately  20  minutes  before  being  closed.  This  yields  a  fully  burdened  per- 
discrepancy  cost,  not  including  equipment  or  materials,  of: 
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xfiZLls  $4,300 
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Thus,  the  estimated  annual  labor  cost  of  handling  discrepancies  for  the  spacecraft  in  this 
study  ranged  from  $390K  to  $1.16M.  It  is  important  to  note  that  this  estimate  does  not 
include  the  cost  of  equipment  or  materials,  which  can  be  considerable  when  the  resolution  of 
a  discrepancy  requires  the  purchase  or  replacement  of  a  major  component  like  a  workstation, 
server,  front-end  processor,  or  ground  station  antenna  motor. 
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Discussion 

Sampling  and  Methodology 

The  true  population  of  interest  for  a  discrepancy  data  analysis  is  the  set  of  all  discrepancies 
occurring  on  all  spacecraft  launched  since  Sputnik.  Obviously,  it  is  not  possible  to 
characterize  the  entire  population  directly,  so  some  sampling  technique  is  required.  This 
study  concentrated  on  spacecraft  that  were  either  currently  operational  or  recently 
decommissioned,  as  of  May  2001. 

The  sample  is  not  truly  representative  of  the  entire  population,  since  it  is  weighted  solely  on 
operations  within  the  last  decade  and  does  not  take  into  account  earlier  missions.  However, 
recently  flown  spacecraft  are  more  relevant,  and  therefore  of  more  interest,  to  existing  and 
future  satellite  missions.  Although  older  discrepancy  data  would  provide  a  more 
comprehensive  dataset,  in  many  cases  operator  logs  were  not  archived  or  are  no  longer 
available.  Therefore,  recent  programs  were  the  only  ones  used  in  this  study. 

The  correlation  analyses  performed  to  test  the  second  hypothesis  indicate  that  correlations  do 
exist  between  certain  design  elements  and  the  types  of  problems  experienced  during 
operations.  Such  analyses  require  that  the  sample  set  be  large  enough  and  diverse  enough  to 
differentiate  the  effects  of  one  design  element  from  the  effects  of  the  others.  For  example,  if 
“orbit  characteristic”  is  the  design  element  of  interest  in  a  particular  analysis,  it  is  desirable  to 
have  every  possible  value  for  “orbit  characteristic”  represented  by  several  spacecraft  in  the 
sample.  This  would  make  it  more  likely  to  identify  a  trend,  if  one  exists.  However,  the 
sample  set  used  in  this  study  did  not  fully  explore  all  of  the  feasible  combinations  of  values 
for  all  of  the  design  elements.  In  some  cases,  like  the  “attitude  control  scheme”  design 
element,  the  sample  set  was  too  uniform  to  represent  the  possible  types  of  attitude  control 
schemes.  The  results  of  several  analyses  were  discarded  due  to  this  situation.  The  problem 
can  be  mitigated  in  the  future  by  more  aggressively  selecting  spacecraft  to  include  in  the 
sample  set,  specifically  focusing  on  spacecraft  which  fully  explore  the  design  element  trade 
space. 
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Observations  on  Data  Collected 

The  process  of  merging  data  originally  recorded  using  different  techniques  requires  care  and 
attention.  The  procedures  for  reporting  problems  vary  from  organization  to  organization,  and 
the  detail  and  content  of  reports  can  even  vary  from  individual  to  individual  using  the  same 
procedure  within  a  single  organization.  In  most  cases,  reviewing  and  categorizing  each 
discrepancy  report  manually  can  overcome  these  variations,  and  was  the  approach  used  in 
this  study. 

Per  Table  5,  software  glitches  were  identified  as  the  root  cause  in  an  average  of  48%  of 
discrepancies  reported  for  each  spacecraft.  Of  all  the  discrepancies  caused  by  software,  9% 
were  attributed  to  spacecraft  components  and  91%  were  attributed  to  infrastructure 
components.  It  is  relevant  to  note  that  most  of  the  satellite  programs  included  in  this  study 
used  commercial  off-the-shelf  (COTS)  software  packages  and/or  hardware  for  data  routing, 
telemetry  processing,  and  command  &  control,  or  had  transitioned  from  a  legacy  system  to  a 
COTS-based  system  at  some  point  during  the  discrepancy  reporting  period. 

Faulty  equipment  was  found  to  be  the  root  cause  in  an  average  of  9%  of  discrepancies 
reported  for  each  spacecraft.  A  review  of  the  corresponding  discrepancy  reports  shows  that 
equipment  problems  are  most  frequently  attributed  to  computer  hard  drive  failures,  processor 
fan  failures,  damaged  cables,  faulty  ground  antenna  amplifier  equipment,  and  facility 
power/utility  equipment. 


Conclusions 

This  study  intended  to  test  the  following  hypotheses: 

•  First  hypothesis.  Most  problems  encountered  by  an  operator  do  not  involve  the 
spacecraft  at  all,  but  are  attributed  to  other  elements  of  the  space  system. 

•  Second  hypothesis.  Correlations  exist  between  aspects  of  a  space  system  design  and 
the  nature  of  problems  experienced  by  the  operations  staff  over  the  long  term. 
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An  analysis  of  over  9,200  discrepancy  reports  from  1 1  on-orbit  spacecraft  supports  the  first 
hypothesis  -  87%  of  the  discrepancies  collected  were  attributed  to  some  component  of  the 
operational  infrastructure.  The  remaining  13%  involved  one  or  more  components  on  board 
the  spacecraft.  Software  was  the  most  frequently  reported  cause  of  discrepancies,  found  in 
61%  of  all  discrepancies  documented. 

The  discrepancy  reports  also  indicated  that  correlations  do  exist  between  certain  design 
elements  and  the  types  of  problems  experienced  during  operations.  The  following 
correlations  were  found  based  on  the  data  collected: 

•  Ground  System  vs.  Mission  Type.  The  percentage  of  discrepancies  per  spacecraft 
associated  with  the  ground  system  tends  to  change  given  a  particular  mission  type  for 
the  spacecraft. 

•  Comm  Relay  vs.  Ops  Team.  The  percentage  of  discrepancies  per  spacecraft 
associated  with  the  communications  relay  tends  to  change  from  one  organization  to 
another. 

Thus,  the  data  collected  supports  the  second  hypothesis,  but  with  the  caveat  that  a  sufficiently 
large  and  diverse  sample  set  must  be  obtained  to  verify  the  results.  It  should  be  noted  that 
causality  cannot  be  determined  from  the  statistical  correlation  analysis,  but  must  be 
investigated  on  a  case-by-case  basis. 

The  results  of  this  study  can  be  extended  by  incorporating  discrepancy  data  from  additional 
spacecraft,  particularly  commercial  programs.  The  methodology  can  also  be  applied  on 
databases  for  individual  satellite  programs  to  gain  insight  into  the  nature  and  frequency  of 
problems  experienced  by  the  operations  staff.  Ultimately,  this  can  help  supervisors  identify 
strengths  and  areas  for  improvement  in  attempt  to  continuously  improve  the  service  provided 
to  the  user. 
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